Commit Graph

67 Commits

Author SHA1 Message Date
James Agnew 7a39f8cbaf
Make reference extractor more reusable (#4205)
* Refactor

* Make reference extractor more reusable

* Test fix

* Test fix

* Headers

* Small refactor
2022-10-27 21:31:39 -04:00
Tadgh 78ce2a6344
4182 convert $mdm submit to a batch job (#4188)
* Changelog

* Wip implementation

* Wip

* wip

* wip

* Fix validator

* Fix up implementation

* tidfy

* TIdy up tests

* Add some docs

* Add changelog info:

* Tidy up self review

* License files

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Ken Stevens <khstevens@gmail.com>

* Review changes

* fixes

* add new method

* Update tests

* Add batch size parameter

Co-authored-by: Ken Stevens <khstevens@gmail.com>
2022-10-25 12:02:04 -07:00
jmarchionatto 3d9dfd4f08
Fix ForcedId criteria query (#4195)
* Fix ForcedId criteria query

* wip parameterizing tests

* Refactor almost all APIs to use ResourcePersistentId. Convert to fhir resource id when necessary (at search time)

* run tests in ANY id mode

* Test fix

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
Co-authored-by: jamesagnew <jamesagnew@gmail.com>
2022-10-25 13:56:08 +00:00
James Agnew a71b8fad56
Add R4B Support (#4150)
* Add r4b stuff

* Work on R4B

* Core version bump

* Revert some changes

* Add R4B

* Docs

* Test fixes

* Work on R4B

* Tests passgin

* Add config

* Work on merging

* Work

* Work

* Work

* Test fix

* Fixes

* Test fixes

* Test fix

* Work

* Work

* Fixes

* Test fixes

* Fixes

* Test fix

* Test fix

* Test fix

* Version bump

* Test fix

* Test fix

* ompile fixes

* Fix

* Resolve build warnings

* Updates

* Test fixes

* Fix docs
2022-10-20 13:57:06 -07:00
Luke deGruchy c8312662d7
Set batch export job results count to number of resources processed instead of 0. (#4172)
* Add count to batch export admin page results and new add new unit test.

* Fix failing unit test in WriteBinaryStepTest.

* Add changelog.

* Small changelog fix.
2022-10-20 14:27:01 -04:00
KGJ-software 69d55b7184
3052 Changed IBinaryStorageSvc MaximumBinarySize from int to long. (#4123)
* 3052 Changed IBinaryStorageSvc MaximumBinarySize from int to long.

* 3052 Added changelog

* 3052 Bumped version

* 3052 Remove mysterious +1 on maximum binary size causing overflow.

* Bumping version to 6.2.0-PRE16-SNAPSHOT

* Adding + 1L as original implementation.

* Getting all tests to pass dealing with + 1L.

* Getting all tests to pass dealing with + 1L.

* Adding migration definition and removing unnecessary comment.

* Preventing register overflow when setting maxBinaryStorage.

* Mering relevant test.

* Fixing typo in migration script

Co-authored-by: kylejule <kyle.jule@smilecdr.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
2022-10-19 06:09:02 -04:00
James Agnew 422ef87a5d
Standardize on one Transactional annotation (#4130)
* Standardize on one Transactional annotation

* Add changelog

* Disable checkstyle rule

* Fix up transactions

* Test fixes

* Test cleanup

* License headers

* One more dep bump

* Resolve commit issue

* Test fixes

* Test fix

* Dep bump

* Fix

* Fix test

* Version bump

* Test fixes
2022-10-13 16:26:11 -04:00
Qingyixia 9d7829a404
Implemented a new setting for Bulk Export that changes how many resources are stored in each binary file (#4119)
* Implemented a new setting for Bulk Export that changes how many resources are stored in each binary file

* Addressing suggestions

* Fixing test

* Fixing test
2022-10-12 10:03:54 -04:00
Luke deGruchy 5831cbedf5
Concurrent cascade delete operations leading to confusing errors (#4126)
* Add unit tests and supporting classes from gitlab issue.

* Commit with changes thus far to be reviewed.   Namely, add savepoint capabilities to HapiTransactionService, inject HapiTransactionService into CascadingDeleteInterceptor, add savepoint capabilities to the TransactionDetails, and catch an Exception in BaseHapiFhirResourceDao.delete() then rollback to the checkpoint.

* Fix compile error.

* hapi transaction service test

* hapi transaction service test

* hapi transaction service test

* First draft of RetryTemplate and flush() logic.  Try unit testing with PointcutLatch.

* hapi transaction service test

* Latest changes including moving RetryTemplate, cleaning up unit tests.

* First attempt at correct design.  Fix compile errors.

* Add failed rollback savepoint test.

* fix propogation

* interesting test failure in SafeDeleterTest

* Latest code

* move delete

* test passing

* all tests pass

* yay we're done just cleanup

* First round of cleanup:  Get rid of all exploratory logging, remove unneeded dependencies, move creation of RetryTemplate to SafeDeleter.

* More cleanup.  Add changelog.

* Bump to 6.2.0-PRE13-SNAPSHOT.

* Code review feedback:  Refactor SafeDeleter to a service and bean.  Add comments.

* Code review feedback:  Comments on unit test assertions.

* Code review feedback:  Rename SafeDeleterSvc to ThreadSafeResourceDeleterSvc.  Add javadoc.  Add logger to DelayListener.   Remove useless commented out code.

* Fix unit test failure in CascadingDeleteInterceptorTest.

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-10-11 16:48:01 -04:00
Ken Stevens 40163f73d2
Ks 20220916 remove flyway (#4040)
* removed flyway, started coding persistence

* started wiring up new dao service

* replace jpa with sqlbuilder

* down to 16 failed tests

* remove outOfOrder test.  We no longer support validating order.

* Removed noFlyway mode.  It is no longer a thing.

* merge migrator

* down to 1 failing test

* cleanup

* removed unused noflyway cli arguments

* cleanup

* cleanup

* cleanup

* cleanup

* create MigrationTaskList to replace the List<BaseTask> that gets passed around a lot.  This allows us to add a bunch of behaviour to that as a first-order thing

* fix test

* fix test

* improve migration table detection to use the catalogue

* Msg.code()

* cleanup

* save version in same format as flyway

* cleanup

* add migration result to migration result class

* failed migrations stay in the table and don't really affect anything.

* change log

* docs

* ugh Flyway requires lower-case column names

* integration test fixes

* log

* fix oracle issue

* lowercase tablenames for postgres.  fussy fussy!

* Revert "lowercase tablenames for postgres.  fussy fussy!"

This reverts commit 92c9cb263b.

* revert to capital table names.  surround table name by quotes for Postgres.

* added log message before executing find all query

* changed log message from info to debug

* javadoc

* pom cleanup

* review feedback

* Update hapi-fhir-sql-migrate/src/main/java/ca/uhn/fhir/jpa/migrate/MigrationTaskList.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* review feedback

* review feedback.  Use prepared statement.

* Change installedOn insert from timestamp to date

* Change installedOn column type from timestamp to date

* bump hapi version

* fix poms

* fix poms and cdr compile

* add exclusions to clean up build

Co-authored-by: Ken Stevens <ken@smilecdr.com>
Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>
2022-10-03 22:40:09 -04:00
weiping202209 66baa8263d
Fix bug in history search (#4009)
* add failing test

* Fix history search bug and add tests for two more scenarios

* Fix history search bug and add tests for two more scenarios

* - added failing test condition

* Fix history search bug and add tests for two more scenarios

* Fix build test failures

* Fix build test failures

* Add extra verifications for scenarios of _since

* Refactor dao.history interface to add overload method and shall come back to remove old method once cdr is refactored.

* Update with new hapi interface

* Add depreciate comment

* Add depreciate comment

* Version bump to mvn 6.2.0-PRE11-SNAPSHOT

Co-authored-by: nathaniel.doef <nathaniel.doef@smilecdr.com>
2022-10-03 09:00:41 -04:00
Ken Stevens e9a855d3f6
remove spring batch (#4064)
* remove spring batch
it compiles
expect test failures

* fix test transaction manager bean name (wow that was quite the hunt the wumpus!)

* fix test and fix build

* fix test and fix build

* fix test

* remove spring batch schema initialization

* bump hapi version

* fix schema migrator

* bump to PRE-10

* merge master

* change logs

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-09-30 13:18:48 -04:00
Tadgh b472c4bb54
4083 batch permissions (#4095)
* Move threadpool factory into a bean

* Move threadpool factory into a bean

* change return type

* Add license files

* Version bump

* Fix test
2022-09-29 17:24:05 +00:00
James Agnew fc82ed4e44
One more tweak to #4093 (#4094)
* One more tweak to #4093

* Add changelog
2022-09-28 16:46:00 +00:00
James Agnew e4a2285f30
Make sure to not mix resource types in bulk export (#4093)
* Make sure to not mix resource types in bulk export

* Avoid dupes

* Add test
2022-09-28 15:16:15 +00:00
StevenXLi 9e1120a47a
4080 support onc requirements for group bulk export (#4081)
* added failing tests

* added include to bulk export processor, added support for different resource types in bulk export fetch id step

* added changelog

* fixed duplication issue, and added null check

Co-authored-by: Steven Li <steven@smilecdr.com>
2022-09-27 14:51:43 +00:00
James Agnew 917cf8d062
Remove ActionRequestDetails (#3527)
* Add test

* Add test

* Start removing unneeded things

* Fixes

* Add changelog

* Test fix

* Address LGTM issues

* Resolve fixme

* Address review comment

* Add changelog

* Update doc

* Test fixes
2022-09-25 21:29:08 +00:00
James Agnew 3bab3544ec
Reduce SQL queries for _include (#4007)
* Rework includes processing for reuse

* Optimize include

* Test fixes

* Bump version down

* Fixed

* Update

* Test fixes

* Start to add javadoc

* Test fixes
2022-09-25 15:22:06 -04:00
James Agnew 3935b78083
Add operation blocking interceptor (#4057)
* Add operation blocking interceptor

* Work on interceptor

* Work on interceptor

* FIx up docs

* Address CVE

* Tweak changelog

* Address review comments
2022-09-25 14:19:17 -04:00
Tadgh 5a978646c5
Add changelog, test, and implementation (#4055) 2022-09-21 21:18:12 -07:00
MykolaMedynskyiSCDR 3349ad8ca9
3929 batch2 jobs file fetching step strict policy for response header content type (#3945)
* Adding failing tests cases.

* What was done:
-Accepting text/plain and application/json as import response content type.
-Added change log.

* Fixed NDJSON EncodingEnum

* Added content type application/json+fhir to the test

* Fixed code

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/6_2_0/3929-batch2-jobs-file-fetching-step-strict-policy-for-response-header-content-type.yaml

Co-authored-by: Tadgh <garygrantgraham@gmail.com>

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/6_2_0/3929-batch2-jobs-file-fetching-step-strict-policy-for-response-header-content-type.yaml

Co-authored-by: Tadgh <garygrantgraham@gmail.com>

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/6_2_0/3929-batch2-jobs-file-fetching-step-strict-policy-for-response-header-content-type.yaml

Co-authored-by: Tadgh <garygrantgraham@gmail.com>

* Changed message

* Fixed code

Co-authored-by: peartree <etienne.poirier@smilecdr.com>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
2022-09-21 08:29:32 -07:00
StevenXLi 3b781022c1
4030 bulk export to support fhir basepatientid$export endpoint (#4032)
* Refactored test helper method to be more generic, added new failing test for feature, and changed BulkDataExportOptions to fix compile error

* added field to all objects needed in the bulk export pipeline for adding patient id, added new operation for the feature

* added changelog

* added support for patient parameter to patient export, changed patient id to a list to match the requirements for the patient parameter, added test for multiple id patient export

* refactored methods, added new changes to changelog

* code review changes

* fix test error

Co-authored-by: Steven Li <steven@smilecdr.com>
2022-09-20 10:42:55 -07:00
Tadgh c3339f38c3
Rel 6 1 2 mergeback (#4033)
* Bump for CVE (#3856)

* Bump for CVE

* Bump spring-data version

* Fix compile

* Cut over to spring bom

* Bump to RC1

* remove RC

* do not contrain reindex for common SP updates (#3876)

* only fast-track jobs with exactly one chunk (#3879)

* Fix illegalstateexception when an exception is thrown during stream response (#3882)

* Finish up changelog, minor refactor

* reset buffer only

* Hack for some replacements

* Failure handling

* wip

* Fixed the issue (#3845)

* Fixed the issue

* Changelog modification

* Changelog modification

* Implemented seventh character extended code and the corresponding dis… (#3709)

* Implemented seventh character extended code and the corresponding display

* Modifications

* Changes on previous test according to modifications made in ICD10-CM XML file

* Subscription sending delete events being skipped (#3888)

* fixed bug and added test

* refactor

* Update for CVE (#3895)

* updated pointcuts to work as intended (#3903)

* updated pointcuts to work as intended

* added changelog

* review fixes

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* 3904 during $delete expunge job hibernate search indexed documents are left orphaned (#3905)

* Add test and implementation

* Add changelog

* 3899 code in limits (#3901)

* Add implementation, changelog, test

* Update hapi-fhir-jpaserver-test-utilities/src/test/java/ca/uhn/fhir/jpa/provider/r4/ResourceProviderR4Test.java

Co-authored-by: Ken Stevens <khstevens@gmail.com>

Co-authored-by: Ken Stevens <khstevens@gmail.com>

* 3884 overlapping searchparameter undetected rel 6 1 (#3909)

* Applying all changes from previous dev branch to current one pointing to rel_6_1

* Fixing merge conflict related to Msg.code value.

* Fixing Msg.code value.

* Making checkstyle happy.

* Making sure that all tests are passing.

* Passing all tests after fixing Msg.code

* Passing all tests.

Co-authored-by: peartree <etienne.poirier@smilecdr.com>

* 3745 - fixed NPE for bundle with duplicate conditional create resourc… (#3746)

* 3745 - fixed NPE for bundle with duplicate conditional create resources and a conditional delete

* created unit test for skip of delete operation while processing duplicating create entries

* moved unit test to FhirSystemDaoR4Test

* 3379 mdm fixes (#3906)

* added MdmLinkCreateSvcimplTest

* fixed creating mdm-link not setting the resource type correctly

* fixed a bug where ResourcePersistenceId was being duplicated instead of passed on

* Update hapi-fhir-jpaserver-mdm/src/test/java/ca/uhn/fhir/jpa/mdm/svc/MdmLinkCreateSvcImplTest.java

Change order of tests such that assertEquals takes expected value then actual value

Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>

* added changelog, also changed a setup function in test to beforeeach

Co-authored-by: Long Ma <long@smilecdr.com>
Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>

* Fix to the issue (#3855)

* Fix to the issue

* Progress

* fixed the issue

* Addressing suggestions

* add response status code to MethodOutcome

* Addressing suggestions

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Fix for caching appearing broken in batch2 for bulkexport jobs (#3912)

* Respect caching in bullk export, fix bug with completed date on empty jobs

* add changelog

* Add impl

* Add breaking test

* Complete failing test

* more broken tests

* Fix more tests'

* Fix paging bug

* Fix another brittle test

* 3915 do not collapse rules with filters (#3916)

* do not attempt to merge compartment permissions with filters

* changelog

* Rename to IT for concurrency problems

Co-authored-by: Tadgh <garygrantgraham@gmail.com>

* Version bump

* fix $mdm-submit output (#3917)

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Gl3407 bundle offset size (#3918)

* begin with failing test

* fixed

* change log

* rollback default count change and corresponding comments

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Offset interceptor now only works for external calls

* Initialize some beans (esp interceptors) later in the boot process so they don't slow down startup.

* do not reindex searchparam jobs on startup

* Fix oracle non-enterprise attempting online index add (#3925)

* 3922 delete expunge large dataset (#3923)

* lower batchsize of delete requests so that we do not get sql exceptions

* blah

* fix test

* updated tests to not fail

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* add index

* Fix up colun grab

* Revert offset mode change

* Revert fix for null/system request details checks for reindex purposes

* Fix bug and add test for SP Validating Interceptor (#3930)

* wip

* Fix uptests

* Fix index online test

* Fix SP validating interceptor logic

* Updating version to: 6.1.1 post release.

* fix compile error

* Deploy to sonatype (#3934)

* adding sonatype profile to checkstyle module

* adding sonatype profile to tinder module

* adding sonatype profile to base pom

* adding final deployToSonatype profile

* wip

* Revert version enum

* Updating version to: 6.1.1 post release.

* Add test, changelog, and implementation

* Add backport info

* Create failing test

* Implemented the fix, fixed existing unit tests

* added changelog

* added test case for no filter, exclude 1 patient

* wip

* Add backport info

* Add info of new version

* Updating version to: 6.1.2 post release.

* bump info and backport for 6.1.2

* Bump for hapi

* Implement bug fixes, add new tests (#4022)

* Implement bug fixes, add new tests

* tidy

* Tidy

* refactor for cleaning

* More tidying

* Lower logging

* Split into nested tests, rename, add todos

* Typo

* Code review

* add backport info

* Tidy versionenum

* Fix import

* Remove files that were repackaged

Co-authored-by: JasonRoberts-smile <85363818+JasonRoberts-smile@users.noreply.github.com>
Co-authored-by: Qingyixia <106992634+Qingyixia@users.noreply.github.com>
Co-authored-by: TipzCM <leif.stawnyczy@gmail.com>
Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <khstevens@gmail.com>
Co-authored-by: Etienne Poirier <33007955+epeartree@users.noreply.github.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
Co-authored-by: kateryna-mironova <107507153+kateryna-mironova@users.noreply.github.com>
Co-authored-by: longma1 <32119004+longma1@users.noreply.github.com>
Co-authored-by: Long Ma <long@smilecdr.com>
Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
Co-authored-by: markiantorno <markiantorno@gmail.com>
Co-authored-by: Steven Li <steven@smilecdr.com>
2022-09-15 22:18:32 +00:00
Tadgh 4226cae752
Broken bulk export (#4012)
* Implement test

* wip

* Fixed a bug in bulk export

* add changelog

* typo

* Switch out to debug logs
2022-09-09 23:18:30 +00:00
jdar8 c647154777
Everything operation with type param code review fixes (#3926)
* check for valid resource type

* add test and refactor test class

* refactor auto get method name

* introduce parameter object to patient everything methods

* introduce parameter object to patient everything methods

* change record to class

* add hapi error code

* refactor

* refactor

* change code num

* refactor getmethodname to util method

* add docs and rename fields to conform to fhir standard

* typo

* version bump

* resolving merge conflicts.

* Bumping version

* Bumping version to 6.2.0-PRE5-SNAPSHOT

* Encapsulating parameters in a PatientEverythingParameters object.

Co-authored-by: Justin_Dar <justin.dar@smilecdr.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
2022-09-07 06:44:36 -04:00
Ken Stevens 100b8f9190
batch2 rework (#3964)
* consolidated Batch2Helper
check status in assert before running maintenance
ignore illegal job instance state transition

* Fix mock tests

* rename intermittent to IT

* extra logging

* into to debug

* review feedback

* added more logging

* removed fasttracking.  replaced with support for triggering a maintenance run

* fixed tests

* fixed tests

* add asserts

* licenses

* snakeyaml cve version bump

* data migration

* merge origin/master

* merge origin/master

* fix test (support triggering jobs when scheduling is disabled)

* add counter to abort potential infinite loop after 100000 calls

* compile error

* move multipartition mdm tests out to a separate IT
update mongo job instance creation with new logic

* fix race condition

* back out infinite loop circuit breaker.  it breaks terminology tests that depend on this loop running > 100000 times.

* fix test (triggered job key was missing group)

* revert saveAllDeferred

* Almost finished getting JpaPersistenceR4IT to pass

* testBulkExportDeleteForOnGoingJob still not passing

* testBulkExportDeleteForOnGoingJob still not passing

* Removed method `fetchInstanceAndMarkInProgress`.  It was clobbering other state transitions like CANCELLED.

* undo rename of cancelled to cancelRequested.  Thymeleaf depends on the old name

* ExpandResourcesStep requires a synchronous search

* Tests are passing but man testBulkExportDeleteForOnGoingJob is a flakey test

* fix test

* fix test

* change log

* allow ERRORED -> FAILED state transition

* add a timeout to saveAllDeferred()

* improve error message on timeout message

* fix tests

* added more logging

* added more logging

* fix semaphore issue

* Msg.code

* fix test

* javadoc

* switch all batch loggers to batch troubleshooting log

* add unit test, improve logging

* move TimeoutManager to its own class.  Add tests.

* fix tests

* comment cleanup

* test race condition

* review feedback

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-09-05 00:04:54 -04:00
StevenXLi 65d4fe81ad
3947 support http delete for operation $export poll status (#3961)
* Added failing test, and the methods that are necessary to not break the build

* Implemented solution to cancel jobs via delete request for poll status

* change log added

* change log name change to pass test

* Changed implementation to work with existing infrastructure

* code review changes

Co-authored-by: Steven Li <steven@smilecdr.com>
2022-08-30 11:30:03 -04:00
Tadgh e7a4c49aac
Rel 6 1 mergeback (#3927)
* Bump for CVE (#3856)

* Bump for CVE

* Bump spring-data version

* Fix compile

* Cut over to spring bom

* Bump to RC1

* remove RC

* do not contrain reindex for common SP updates (#3876)

* only fast-track jobs with exactly one chunk (#3879)

* Fix illegalstateexception when an exception is thrown during stream response (#3882)

* Finish up changelog, minor refactor

* reset buffer only

* Hack for some replacements

* Failure handling

* wip

* Fixed the issue (#3845)

* Fixed the issue

* Changelog modification

* Changelog modification

* Implemented seventh character extended code and the corresponding dis… (#3709)

* Implemented seventh character extended code and the corresponding display

* Modifications

* Changes on previous test according to modifications made in ICD10-CM XML file

* Subscription sending delete events being skipped (#3888)

* fixed bug and added test

* refactor

* Update for CVE (#3895)

* updated pointcuts to work as intended (#3903)

* updated pointcuts to work as intended

* added changelog

* review fixes

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* 3904 during $delete expunge job hibernate search indexed documents are left orphaned (#3905)

* Add test and implementation

* Add changelog

* 3899 code in limits (#3901)

* Add implementation, changelog, test

* Update hapi-fhir-jpaserver-test-utilities/src/test/java/ca/uhn/fhir/jpa/provider/r4/ResourceProviderR4Test.java

Co-authored-by: Ken Stevens <khstevens@gmail.com>

Co-authored-by: Ken Stevens <khstevens@gmail.com>

* 3884 overlapping searchparameter undetected rel 6 1 (#3909)

* Applying all changes from previous dev branch to current one pointing to rel_6_1

* Fixing merge conflict related to Msg.code value.

* Fixing Msg.code value.

* Making checkstyle happy.

* Making sure that all tests are passing.

* Passing all tests after fixing Msg.code

* Passing all tests.

Co-authored-by: peartree <etienne.poirier@smilecdr.com>

* 3745 - fixed NPE for bundle with duplicate conditional create resourc… (#3746)

* 3745 - fixed NPE for bundle with duplicate conditional create resources and a conditional delete

* created unit test for skip of delete operation while processing duplicating create entries

* moved unit test to FhirSystemDaoR4Test

* 3379 mdm fixes (#3906)

* added MdmLinkCreateSvcimplTest

* fixed creating mdm-link not setting the resource type correctly

* fixed a bug where ResourcePersistenceId was being duplicated instead of passed on

* Update hapi-fhir-jpaserver-mdm/src/test/java/ca/uhn/fhir/jpa/mdm/svc/MdmLinkCreateSvcImplTest.java

Change order of tests such that assertEquals takes expected value then actual value

Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>

* added changelog, also changed a setup function in test to beforeeach

Co-authored-by: Long Ma <long@smilecdr.com>
Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>

* Fix to the issue (#3855)

* Fix to the issue

* Progress

* fixed the issue

* Addressing suggestions

* add response status code to MethodOutcome

* Addressing suggestions

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Fix for caching appearing broken in batch2 for bulkexport jobs (#3912)

* Respect caching in bullk export, fix bug with completed date on empty jobs

* add changelog

* Add impl

* Add breaking test

* Complete failing test

* more broken tests

* Fix more tests'

* Fix paging bug

* Fix another brittle test

* 3915 do not collapse rules with filters (#3916)

* do not attempt to merge compartment permissions with filters

* changelog

* Rename to IT for concurrency problems

Co-authored-by: Tadgh <garygrantgraham@gmail.com>

* Version bump

* fix $mdm-submit output (#3917)

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Gl3407 bundle offset size (#3918)

* begin with failing test

* fixed

* change log

* rollback default count change and corresponding comments

Co-authored-by: Ken Stevens <ken@smilecdr.com>

* Offset interceptor now only works for external calls

* Initialize some beans (esp interceptors) later in the boot process so they don't slow down startup.

* do not reindex searchparam jobs on startup

* Fix oracle non-enterprise attempting online index add (#3925)

* 3922 delete expunge large dataset (#3923)

* lower batchsize of delete requests so that we do not get sql exceptions

* blah

* fix test

* updated tests to not fail

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* add index

* Fix up colun grab

* Bump Version for non-breaking purposes

* Fix broken test

* bump error code

* revert internal logic

* Revert offset mode change

* wip

* Revert fix for null/system request details checks for reindex purposes

* Fix bug and add test for SP Validating Interceptor (#3930)

* wip

* Fix uptests

* Fix index online test

* Fix SP validating interceptor logic

Co-authored-by: JasonRoberts-smile <85363818+JasonRoberts-smile@users.noreply.github.com>
Co-authored-by: Qingyixia <106992634+Qingyixia@users.noreply.github.com>
Co-authored-by: TipzCM <leif.stawnyczy@gmail.com>
Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <khstevens@gmail.com>
Co-authored-by: Etienne Poirier <33007955+epeartree@users.noreply.github.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
Co-authored-by: kateryna-mironova <107507153+kateryna-mironova@users.noreply.github.com>
Co-authored-by: longma1 <32119004+longma1@users.noreply.github.com>
Co-authored-by: Long Ma <long@smilecdr.com>
Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-08-18 17:21:27 -07:00
michaelabuckley 60a879c842
Renable elasticsearch tests (#3850)
* Renable elasticsearch tests

Limit elasticsearch container to 512MB instead of the default 4GB.
2022-08-07 07:09:12 +00:00
TipzCM eae1c0ffb8
3846 migrate bulk import pull to batch2 (#3853)
* still fixing tests

* migrated a bulk import pull job t obatch 2

* changelog

* cleanup

* review fixes

* minor tweaks

* cleaningup

* tra la la

* blah

* blah

* updating msg

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
2022-08-03 10:50:51 -04:00
Tadgh dcfaf42e73 add new version enum, bump version, new folder 2022-08-02 21:46:20 -07:00
jdar8 5f1c50a6b8
everything operation with type param (#3842)
* wip+test

* wip, rough implementation

* clean up

* clean up

* clean up unused methods

* Completing implementation.

* Removing TODO.

* Replacing toList() with Collectors.toList()

* support other fhir versions and everything instance

* tests wip

* tests for instance

* Adding default methods in interface

* add testsss

* Removing @Deprecated.

* fix spacing

* changelog

* reword changelog

* Removing default methods from interface.

* Bumping version to 6.1.0-PRE28-SNAPSHOT

* Adding parameter to method call following interface modification.

* Fixing documentation and parameter order for consistency.

* Bumping version to PRE30.

* Providing backward compatibility.

* fix pom

* licences

* Fixing pom dependencies.

* Fixing more pom dependencies.

* Fixing dependencies for test-utilities.

Co-authored-by: Justin_Dar <justin.dar@smilecdr.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-08-02 20:51:37 -07:00
Nathan Doef e2bed96d74
3776 add https support for hapi fhir cli using tls (#3777)
* wip functional
TODO
- add tests
- look at adding TLS to other RestfulClientFactory classes (FhirwebRestfulClientFactory, JaxRsRestfulClientFactory, OkHttpRestfulClientFactory)

* move BaseRestServerHelper to Hapi-Fhir

* - add TLS support for JaxRsClient and OkHttpClient
- add tests

* -  to use resources and files

* -  ci fixes

* -  ci fixes

* -  ci fixes

* -  ci fixes

* - refactor and tests

* - ci fixes

* - tests

* - tests

* - tests / refactor

* - clean up

* - changelog

* - remove OkHttpClient HTTPS support

* - ci

* - Ken's code review changes

* - Ken's code review: method renaming

* - Ken's code review: javadoc

* - Code coverage

* - Create HapiFhirCliRestfulClientFactory to encapsulate new HTTPS functionality. Remove modifications to the FhirContext API.

* - version bump

* - fix unable to bind port errors by using Jetty functionality to assign random ports

* Revert "- version bump"

This reverts commit cfce82576a.

* - version bump

Co-authored-by: nathaniel.doef <nathaniel.doef@smilecdr.com>
2022-08-02 09:27:02 -04:00
David be0b6f1a0b
3840 add sort parameter to fetching jobs (#3861)
* added a paging api to ijobcoordinator

* adding changelog

* moved location

* added a comment

* Added Sort parameter to JobInstanceFetchRequest

* Bump HAPI-FHIR version

* Updated JPA api to include sort

* Add changelog

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: David Chen <david.chen@smilecdr.com>
2022-07-29 09:12:53 -06:00
longma1 f933ff3d9c
2878 mdm mongo support (#3748)
* small refactor

* refactored MDM module to use iterface IMDMLink, refactored MDMLink to use ResourcePersistenceId wrapper rather than pid as long

* removing IJpaIdHelperService and refactor IIdHelperService

* fixed bugs for GoldenResourceMergerSvcImpl and FindCandidateByExampleSvc

* changes to IIdHelperService

* move MdmSearchParamSvc and MdmSubmitSvcImpl out of jpaserver-mdm

* moved JpaResourceLoader to hapi-fhir-storage, renamed to ResourceLoaderImpl

* added error Msg code, removed Fixme

* fixed error code, update last used code in Msg.java to 2083

* fixed accidentally throwing unprocessableentityException

* added MdmLinkExpandSvc interface

* build adding license to files

* MdmLinkdao refactor

* fix build

* mdm config refactoring

* MdmStorageInterceptor and IExpungeEverythingService changes

* fix bug where a pid is converted to ResourcePersistenceId twice

* minor refactor to IMdmLink, change id from long to ResourcePersistenceId

* changed PageImpl to Page interface

* refactor MdmSearchExpandingInterceptor to use interface

* fixed missed refactor where the resourcePersistenceId is still assumed to be long

* resolve test failures resulting from merging master

* changed conflicting exception code

* fixed issue with bulk export wrapping resourcePid inside resourcePersistenceId

* added IMdmLinkImpl to MdmSettings

* fixed missing import in IdHelperService from merging master

* fixed MdmLinkDaoSvcTest importing wrong MdmLinkTuple

* restored JpaIdHelperService and added changelog

* modified msg.code to non-duplicate

* code review changes

* refactored some bulk export batch 2 files to accomodate change in beans being exported

* fixed jpa bulk export failing tests

* removed duplicate bean declaration of MdmLinkImplFactory in BaseMdmR4Test

* merged master and bump version to pre25

* extracted expungeEverythingByType to IExpungeEverythingService

* fixed wiring issue

* messed up merging pom with master

* fixed broken changelog

Co-authored-by: Long Ma <long@smilecdr.com>
Co-authored-by: Samuel Lee <samuel.lee@smilecdr.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-29 08:58:40 -06:00
jmarchionatto 8346c0263b
Issue 3804 application fails to start 610 pre14 snapshot with beandefinitionoverrideexception invalid bean definition with name loadidsstep (#3819)
* shared steps

* Add changelog

* Rename as suggested

* Remove unnecessary bean

* Add missing imports

* Remove parent class not used anymore

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-07-29 09:23:34 -04:00
TipzCM 9a1d6f3172
3821 term code system batch jobs (#3826)
* step one

* blah

* no working test

* updated existing tests

* delete code system version and code system jobs

* trying to get terminology fixed

* updating tests

* need to have some movement

* added completion handler

* added clearing code for failsafe

* review points

* fixing tests

* fixing tests

* more test fixes

* fixing more tests

* review fixes

* todo

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
2022-07-27 14:50:42 -04:00
TipzCM 59540ce803
3813 paging job coordinator (#3814)
* added a paging api to ijobcoordinator

* adding changelog

* moved location

* added a comment

* blah

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
2022-07-26 11:29:37 -06:00
Ken Stevens 533019f051
Response Terminology Wiring (#3807)
* cleaned up response terminology wiring

* jpa is failing atm

* bump pom versions

* double build RAM

* bump hapi version to PRE-22

* license headers

* re-bump to PRE-22

* tag intermittent

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-25 18:41:01 -04:00
Ken Stevens f01e4a7ad2
Remote Terminology support refactor (#3822)
* Revert "test passes with a lot of copy/paste"

This reverts commit f2f28e22ef.

* make translateOutcomeToResults public

* double RAM

* final consolidation.  move shared code out to static utility methods

* revert azure memory
disable jacoco on jpaserver test prokect

* bump hapi version

* rebump hapi to PRE-21

* licenses

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-25 11:47:31 -04:00
Ken Stevens 611c1d9d2c
enable partition-aware subscription activation (#3844)
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-25 06:59:49 +00:00
Etienne Poirier d2fd0989c3
3820 Providing an exception for search request with invalid or incomplete tokenParam (#3825)
* Creating an exception to be raised when searching for _tag/_provide/_security where the associated tokenParam is not properly formatted.

* Adding changelogs.

* Adding integration test.

* Rewording ChangeLog.

* Rewording exception message.

* Addressing code review comments.

* Addressing code review comments.

* Bumping hapi-fhir version to 6.1.0-PRE20-SNAPSHOT

* Modifications to pass checkstyle.

Co-authored-by: peartree <etienne.poirier@smilecdr.com>
2022-07-22 11:55:44 -04:00
jmarchionatto 67e67bbf44
Issue 3804 application fails to start 610 pre14 snapshot with beandefinitionoverrideexception invalid bean definition with name loadidsstep (#3805)
* shared steps

* Add changelog

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-07-19 10:00:28 -04:00
Qingyixia 592190701c
Implement patch in mongo db (#3781)
* Not the final version

* Support PATCH operation

* minor modifications

* minor modifications

* Final version

* Final version

* Version Bump

* bump version again

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-17 21:26:34 -04:00
TipzCM 8ed0a18ef4
3531 implement bulk processors (#3566)
* base batch2 bulk export steps - unimplemented

* added the basic steps

* adding tests

* we have added the tests

* added tests for bulk export stages

* implementing jpa

* 3531 adding the jpabukexportprocessorimplementation

* 3531 first step for mongo bulk export

* jpa stuff updated

* 3531 hapi fhir stuff

* trying to fix up tests

* fixing test

* fixed most tests in batch export

* deprecate

* need some more guidance here

* fixing some tests

* some steps taken

* blah

* stashing

* test

* fixing tests round 1

* many a changes

* updating version

* small updates

* merging in mster again

* fixing up serialization

* added the permission checkers

* cleanup and sonarkube crap

* some more tests

* updating adding tests

* crimes

* blah

* updatingmsg number again

* removal of commented code

* cleanup

* merge confflicts

* fixing msg

* review fixes round 1

* more review fixes

* merging in master

* fixing poms

* review points

* test fix

* test fix

* fix test race condition

* truncate error

* add stack trace to error

* rename to IT

* fix race condition in test

* finally fixed race condition in test.  this time I'm optimistic hapi ci will finally pass

* fix mock test.  not a fan of mock tests like this.

* any job definition with a reducer step must be gated.

* fixed common batch bean wiring

* fix test

* lazy init ResponseTerminologyTranslationSvc

* change the way we lazy init ResponseTerminologyTranslationSvc

* change the way we lazy init ResponseTerminologyTranslationSvc (for the third time.  this is super tricky!)

* disabling testGoldenResourceWithCircularReferenceCanBeCleared for now

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-17 17:49:03 -04:00
TipzCM 0c644271ce
3770 fix batch2 failure loop (#3779)
* 3770 fixing error handling

* some updates to error handling

* changelog

* blah

* bumping version

* fix test failures

* code review points

* review points

* add retry test

* assert counts in the test

* test fixes

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-15 12:42:46 -04:00
StevenXLi 2bc6c91153
3719 enable map response terminology for bulk export (#3735)
* added failing test for Map Response Terminology for bulk export

* moved logic for terminology translation out of the interceptor to svc, added translation logic to bulk export file writer, added more tests

* added changelog

* Wip refactor dependency fix

* wip terminology translation code refactor

* terminology translation code refactor

* refactor clean up

* Remove circular dependency

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/6_1_0/3719-enable-map-response-terminology-for-bulk-export.yaml

* bump version of hapi to 6.1.0-PRE7-SNAPSHOT

* doc update

* fix issue when search param registry doesn't contain search param for the resource type

* fix beans

* hapi version bump from PRE7 to PRE10

* doc update

* add ResponseTerminologyTranslationSvc as a constructor variable to the interceptor

* fixed ResponseTerminologyInterceptorLoaderTest, added IT for cdr side

* version bump to PRE13

* removed unused import

Co-authored-by: Steven Li <steven@smilecdr.com>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-12 12:24:15 -04:00
Ken Stevens 0526da080b
3757 move delete expunge to batch2 (#3759)
* begin with failing test

* begin with failing test

* remove unused Spring Batch classes.
convert service classes

* move provider

* just one test left to fix

* fix test

* msg.code

* change log

* bumping hapi to PRE12

* fix test

* review feedback

* license update

* fix intermittent and status update atomicity

* restore url-level partition selection

* fix tests

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-09 17:25:55 -04:00
Nathan Doef b68ccb373a
3693 Search by id is returning a deleted resource with a client-generated id (#3694)
- Added functionality to optionally filter out deleted resources when resolving forced ids to persistent ids
- Bumps version

Co-authored-by: nathaniel.doef <nathaniel.doef@smilecdr.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-05 12:37:25 -04:00
Ken Stevens 3a28920ea7
Add error-handler callback and ensure batch2 completion handler can't be called twice (#3752)
* rename field that drives everything includes

* discovered design flaw in completion

* fix intermittent

* ensure batch2 completion handler can't be called twice

* changelog

* changelog

* javadoc

* bump hapi version

* licences

* Batch2 job definitions can now optionally provide an error handler callback that will be called when a job
instance fails, errors or is cancelled.

* Changed error handler and completion handler to take a "read-only" copy of an instance instead of just certain fields.

* comment

* javadoc

* fix test

* change step execution context from taking instance id to instance

* review feedback

* fix test

* replace 20 or so jetbrains annotations

* fixed a few incorrect @NotNull annotations

Co-authored-by: Ken Stevens <ken@smilecdr.com>
2022-07-04 23:27:58 -04:00