Reverse Chaining searches returns an error when invoked with parameter _lastUpdated. (#5177)

* version bump

* Bump to core release 6.0.22 (#5028)

* Bump to core release 6.0.16

* Bump to core version 6.0.20

* Fix errors thrown as a result of VersionSpecificWorkerContextWrapper

* Bump to core 6.0.22

* Resolve 5126 hfj res ver prov might cause migration error on db that automatically indexes the primary key (#5127)

* dropped old index FK_RESVERPROV_RES_PID on RES_PID column before adding IDX_RESVERPROV_RES_PID

* added changelog

* changed to valid version number

* changed to valid version number, need to be ordered by version number...

* 5123 - Use DEFAULT partition for server-based requests if none specified (#5124)

5123 - Use DEFAULT partition for server-based requests if none specified

* consent remove all suppresses next link in bundle (#5119)

* added FIXME with source of issue

* added FIXME with root cause

* added FIXME with root cause

* Providing solution to the issue and removing fixmes.

* Providing changelog

* auto-formatting.

* Adding new test.

* Adding a new test for standard paging

* let's try this and see if it works...?

* fix tests

* cleanup to trigger a new run

* fixing tests

---------

Co-authored-by: Ken Stevens <ken@smilecdr.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>

* 5117 MDM Score for No Match Fields Should Not Be Included in Total Score  (#5118)

* fix, test, changelog

* fix, test, changelog

---------

Co-authored-by: justindar <justin.dar@smilecdr.com>

* _source search parameter needs to support modifiers (#5095)

_source search parameter needs to support modifiers - added support form :contains, :missing, :above modifiers

* Fix HFQL docs (#5151)

* Expunge operation on codesystem may throw 500 internal error with precondition fail message. (#5156)

* Initial failing test.

* Solution with changelog.

* fixing format.

* Addressing comment from code review.

* fixing failing test.

---------

Co-authored-by: peartree <etienne.poirier@smilecdr.com>

* documentation update (#5154)

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* Fix hsql jdbc driver deps (#5168)

Avoid non-included classes in jdbc driver dependencies.

* $delete-expunge over 10k resources will now delete all resources (#5144)

* First commit with very rough fix and unit test.

* Refinements to ResourceIdListStep and Batch2DaoSvcImpl.  Make LoadIdsStepTest pass.   Enhance Batch2DaoSvcImplTest.

* Spotless

* Fix checkstyle errors.

* Fix test failures.

* Minor refactoring.  New unit test.  Finalize changelist.

* Spotless fix.

* Delete now useless code from unit test.

* Delete more useless code.

* Test pre-commit hook

* More spotless fixes.

* Address most code review feedback.

* Remove use of pageSize parameter and see if this breaks the pipeline.

* Remove use of pageSize parameter and see if this breaks the pipeline.

* Fix the noUrl case by passing an unlimited Pegeable instead.  Effectively stop using page size for most databases.

* Deprecate the old method and have it call the new one by default.

* updating documentation (#5170)

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>

* _source search parameter modifiers for Subscription matching (#5159)

* _source search parameter modifiers for Subscription matching - test, implementation and changelog

* first fix

* tests and preliminary fixes

* wip, commit before switching to release branch.

* adding capability to handle _lastUpdated in reverse search (_has)

* adding changelog

* applying spotless.

* addressing code review comments.

---------

Co-authored-by: tadgh <garygrantgraham@gmail.com>
Co-authored-by: dotasek <david.otasek@smilecdr.com>
Co-authored-by: Steve Corbett <137920358+steve-corbett-smilecdr@users.noreply.github.com>
Co-authored-by: Ken Stevens <khstevens@gmail.com>
Co-authored-by: Ken Stevens <ken@smilecdr.com>
Co-authored-by: peartree <etienne.poirier@smilecdr.com>
Co-authored-by: jdar8 <69840459+jdar8@users.noreply.github.com>
Co-authored-by: justindar <justin.dar@smilecdr.com>
Co-authored-by: volodymyr-korzh <132366313+volodymyr-korzh@users.noreply.github.com>
Co-authored-by: Nathan Doef <n.doef@protonmail.com>
Co-authored-by: Etienne Poirier <33007955+epeartree@users.noreply.github.com>
Co-authored-by: TipzCM <leif.stawnyczy@gmail.com>
Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>
Co-authored-by: Luke deGruchy <luke.degruchy@smilecdr.com>
This commit is contained in:
TynerGjs 2023-10-25 08:47:07 -04:00 committed by GitHub
parent 52bdc2693c
commit a40adab882
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 244 additions and 63 deletions

View File

@ -0,0 +1,6 @@
---
type: fix
issue: 5176
jira: SMILE-6333
title: "Previously, the use of search parameter _lastUpdated as part of a reverse chaining search would return an error
message to the client. This issue has been fixed"

View File

@ -69,10 +69,10 @@ import ca.uhn.fhir.parser.DataFormatException;
import ca.uhn.fhir.rest.api.Constants;
import ca.uhn.fhir.rest.api.QualifiedParamList;
import ca.uhn.fhir.rest.api.RestSearchParameterTypeEnum;
import ca.uhn.fhir.rest.api.SearchContainedModeEnum;
import ca.uhn.fhir.rest.api.server.RequestDetails;
import ca.uhn.fhir.rest.param.CompositeParam;
import ca.uhn.fhir.rest.param.DateParam;
import ca.uhn.fhir.rest.param.DateRangeParam;
import ca.uhn.fhir.rest.param.HasParam;
import ca.uhn.fhir.rest.param.NumberParam;
import ca.uhn.fhir.rest.param.QuantityParam;
@ -123,6 +123,7 @@ import java.util.function.Supplier;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import static ca.uhn.fhir.jpa.search.builder.QueryStack.SearchForIdsParams.with;
import static ca.uhn.fhir.jpa.util.QueryParameterUtils.fromOperation;
import static ca.uhn.fhir.jpa.util.QueryParameterUtils.getChainedPart;
import static ca.uhn.fhir.jpa.util.QueryParameterUtils.getParamNameWithPrefix;
@ -1107,7 +1108,7 @@ public class QueryStack {
if (paramName.startsWith("_has:")) {
ourLog.trace("Handing double _has query: {}", paramName);
ourLog.trace("Handling double _has query: {}", paramName);
String qualifier = paramName.substring(4);
for (IQueryParameterType next : nextOrList) {
@ -1160,26 +1161,30 @@ public class QueryStack {
parameterName = parameterName.substring(0, colonIndex);
}
ResourceLinkPredicateBuilder join =
ResourceLinkPredicateBuilder resourceLinkTableJoin =
mySqlBuilder.addReferencePredicateBuilderReversed(this, theSourceJoinColumn);
Condition partitionPredicate = join.createPartitionIdPredicate(theRequestPartitionId);
Condition partitionPredicate = resourceLinkTableJoin.createPartitionIdPredicate(theRequestPartitionId);
List<String> paths = join.createResourceLinkPaths(targetResourceType, paramReference, new ArrayList<>());
List<String> paths = resourceLinkTableJoin.createResourceLinkPaths(
targetResourceType, paramReference, new ArrayList<>());
if (CollectionUtils.isEmpty(paths)) {
throw new InvalidRequestException(Msg.code(2305) + "Reference field does not exist: " + paramReference);
}
Condition typePredicate = BinaryCondition.equalTo(
join.getColumnTargetResourceType(), mySqlBuilder.generatePlaceholder(theResourceType));
Condition pathPredicate =
toEqualToOrInPredicate(join.getColumnSourcePath(), mySqlBuilder.generatePlaceholders(paths));
Condition linkedPredicate = searchForIdsWithAndOr(
join.getColumnSrcResourceId(),
targetResourceType,
parameterName,
Collections.singletonList(orValues),
theRequest,
theRequestPartitionId,
SearchContainedModeEnum.FALSE);
resourceLinkTableJoin.getColumnTargetResourceType(),
mySqlBuilder.generatePlaceholder(theResourceType));
Condition pathPredicate = toEqualToOrInPredicate(
resourceLinkTableJoin.getColumnSourcePath(), mySqlBuilder.generatePlaceholders(paths));
Condition linkedPredicate =
searchForIdsWithAndOr(with().setSourceJoinColumn(resourceLinkTableJoin.getColumnSrcResourceId())
.setResourceName(targetResourceType)
.setParamName(parameterName)
.setAndOrParams(Collections.singletonList(orValues))
.setRequest(theRequest)
.setRequestPartitionId(theRequestPartitionId));
andPredicates.add(toAndPredicate(partitionPredicate, pathPredicate, typePredicate, linkedPredicate));
}
@ -2270,57 +2275,91 @@ public class QueryStack {
}
@Nullable
public Condition searchForIdsWithAndOr(
@Nullable DbColumn theSourceJoinColumn,
String theResourceName,
String theParamName,
List<List<IQueryParameterType>> theAndOrParams,
RequestDetails theRequest,
RequestPartitionId theRequestPartitionId,
SearchContainedModeEnum theSearchContainedMode) {
public Condition searchForIdsWithAndOr(SearchForIdsParams theSearchForIdsParams) {
if (theAndOrParams.isEmpty()) {
if (theSearchForIdsParams.myAndOrParams.isEmpty()) {
return null;
}
switch (theParamName) {
switch (theSearchForIdsParams.myParamName) {
case IAnyResource.SP_RES_ID:
return createPredicateResourceId(
theSourceJoinColumn, theAndOrParams, theResourceName, null, theRequestPartitionId);
theSearchForIdsParams.mySourceJoinColumn,
theSearchForIdsParams.myAndOrParams,
theSearchForIdsParams.myResourceName,
null,
theSearchForIdsParams.myRequestPartitionId);
case PARAM_HAS:
return createPredicateHas(
theSourceJoinColumn, theResourceName, theAndOrParams, theRequest, theRequestPartitionId);
theSearchForIdsParams.mySourceJoinColumn,
theSearchForIdsParams.myResourceName,
theSearchForIdsParams.myAndOrParams,
theSearchForIdsParams.myRequest,
theSearchForIdsParams.myRequestPartitionId);
case Constants.PARAM_TAG:
case Constants.PARAM_PROFILE:
case Constants.PARAM_SECURITY:
if (myStorageSettings.getTagStorageMode() == JpaStorageSettings.TagStorageModeEnum.INLINE) {
return createPredicateSearchParameter(
theSourceJoinColumn,
theResourceName,
theParamName,
theAndOrParams,
theRequest,
theRequestPartitionId);
theSearchForIdsParams.mySourceJoinColumn,
theSearchForIdsParams.myResourceName,
theSearchForIdsParams.myParamName,
theSearchForIdsParams.myAndOrParams,
theSearchForIdsParams.myRequest,
theSearchForIdsParams.myRequestPartitionId);
} else {
return createPredicateTag(theSourceJoinColumn, theAndOrParams, theParamName, theRequestPartitionId);
return createPredicateTag(
theSearchForIdsParams.mySourceJoinColumn,
theSearchForIdsParams.myAndOrParams,
theSearchForIdsParams.myParamName,
theSearchForIdsParams.myRequestPartitionId);
}
case Constants.PARAM_SOURCE:
return createPredicateSourceForAndList(theSourceJoinColumn, theAndOrParams);
return createPredicateSourceForAndList(
theSearchForIdsParams.mySourceJoinColumn, theSearchForIdsParams.myAndOrParams);
case Constants.PARAM_LASTUPDATED:
// this case statement handles a _lastUpdated query as part of a reverse search
// only (/Patient?_has:Encounter:patient:_lastUpdated=ge2023-10-24).
// performing a _lastUpdated query on a resource (/Patient?_lastUpdated=eq2023-10-24)
// is handled in {@link SearchBuilder#createChunkedQuery}.
return createReverseSearchPredicateLastUpdated(
theSearchForIdsParams.myAndOrParams, theSearchForIdsParams.mySourceJoinColumn);
default:
return createPredicateSearchParameter(
theSourceJoinColumn,
theResourceName,
theParamName,
theAndOrParams,
theRequest,
theRequestPartitionId);
theSearchForIdsParams.mySourceJoinColumn,
theSearchForIdsParams.myResourceName,
theSearchForIdsParams.myParamName,
theSearchForIdsParams.myAndOrParams,
theSearchForIdsParams.myRequest,
theSearchForIdsParams.myRequestPartitionId);
}
}
private Condition createReverseSearchPredicateLastUpdated(
List<List<IQueryParameterType>> theAndOrParams, DbColumn theSourceColumn) {
ResourceTablePredicateBuilder resourceTableJoin =
mySqlBuilder.addResourceTablePredicateBuilder(theSourceColumn);
List<Condition> andPredicates = new ArrayList<>(theAndOrParams.size());
for (List<IQueryParameterType> aList : theAndOrParams) {
if (!aList.isEmpty()) {
DateParam dateParam = (DateParam) aList.get(0);
DateRangeParam dateRangeParam = new DateRangeParam(dateParam);
Condition aCondition = mySqlBuilder.addPredicateLastUpdated(dateRangeParam, resourceTableJoin);
andPredicates.add(aCondition);
}
}
return toAndPredicate(andPredicates);
}
@Nullable
private Condition createPredicateSearchParameter(
@Nullable DbColumn theSourceJoinColumn,
@ -3020,4 +3059,82 @@ public class QueryStack {
theParamDefinition, myOrValues, myLeafTarget, myLeafParamName, myLeafPathPrefix, myQualifiers);
}
}
public static class SearchForIdsParams {
DbColumn mySourceJoinColumn;
String myResourceName;
String myParamName;
List<List<IQueryParameterType>> myAndOrParams;
RequestDetails myRequest;
RequestPartitionId myRequestPartitionId;
ResourceTablePredicateBuilder myResourceTablePredicateBuilder;
public static SearchForIdsParams with() {
return new SearchForIdsParams();
}
public DbColumn getSourceJoinColumn() {
return mySourceJoinColumn;
}
public SearchForIdsParams setSourceJoinColumn(DbColumn theSourceJoinColumn) {
mySourceJoinColumn = theSourceJoinColumn;
return this;
}
public String getResourceName() {
return myResourceName;
}
public SearchForIdsParams setResourceName(String theResourceName) {
myResourceName = theResourceName;
return this;
}
public String getParamName() {
return myParamName;
}
public SearchForIdsParams setParamName(String theParamName) {
myParamName = theParamName;
return this;
}
public List<List<IQueryParameterType>> getAndOrParams() {
return myAndOrParams;
}
public SearchForIdsParams setAndOrParams(List<List<IQueryParameterType>> theAndOrParams) {
myAndOrParams = theAndOrParams;
return this;
}
public RequestDetails getRequest() {
return myRequest;
}
public SearchForIdsParams setRequest(RequestDetails theRequest) {
myRequest = theRequest;
return this;
}
public RequestPartitionId getRequestPartitionId() {
return myRequestPartitionId;
}
public SearchForIdsParams setRequestPartitionId(RequestPartitionId theRequestPartitionId) {
myRequestPartitionId = theRequestPartitionId;
return this;
}
public ResourceTablePredicateBuilder getResourceTablePredicateBuilder() {
return myResourceTablePredicateBuilder;
}
public SearchForIdsParams setResourceTablePredicateBuilder(
ResourceTablePredicateBuilder theResourceTablePredicateBuilder) {
myResourceTablePredicateBuilder = theResourceTablePredicateBuilder;
return this;
}
}
}

View File

@ -131,6 +131,7 @@ import javax.persistence.criteria.CriteriaBuilder;
import static ca.uhn.fhir.jpa.model.util.JpaConstants.UNDESIRED_RESOURCE_LINKAGES_FOR_EVERYTHING_ON_PATIENT_INSTANCE;
import static ca.uhn.fhir.jpa.search.builder.QueryStack.LOCATION_POSITION;
import static ca.uhn.fhir.jpa.search.builder.QueryStack.SearchForIdsParams.with;
import static org.apache.commons.lang3.StringUtils.defaultString;
import static org.apache.commons.lang3.StringUtils.isBlank;
import static org.apache.commons.lang3.StringUtils.isNotBlank;
@ -281,14 +282,11 @@ public class SearchBuilder implements ISearchBuilder<JpaPid> {
continue;
}
List<List<IQueryParameterType>> andOrParams = myParams.get(nextParamName);
Condition predicate = theQueryStack.searchForIdsWithAndOr(
null,
myResourceName,
nextParamName,
andOrParams,
theRequest,
myRequestPartitionId,
searchContainedMode);
Condition predicate = theQueryStack.searchForIdsWithAndOr(with().setResourceName(myResourceName)
.setParamName(nextParamName)
.setAndOrParams(andOrParams)
.setRequest(theRequest)
.setRequestPartitionId(myRequestPartitionId));
if (predicate != null) {
theSearchSqlBuilder.addPredicate(predicate);
}

View File

@ -51,7 +51,6 @@ import ca.uhn.fhir.model.primitive.IdDt;
import ca.uhn.fhir.parser.DataFormatException;
import ca.uhn.fhir.rest.api.Constants;
import ca.uhn.fhir.rest.api.RestSearchParameterTypeEnum;
import ca.uhn.fhir.rest.api.SearchContainedModeEnum;
import ca.uhn.fhir.rest.api.server.RequestDetails;
import ca.uhn.fhir.rest.param.ReferenceParam;
import ca.uhn.fhir.rest.param.TokenParam;
@ -87,6 +86,7 @@ import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import static ca.uhn.fhir.jpa.search.builder.QueryStack.SearchForIdsParams.with;
import static org.apache.commons.lang3.StringUtils.isBlank;
import static org.apache.commons.lang3.StringUtils.trim;
@ -456,14 +456,13 @@ public class ResourceLinkPredicateBuilder extends BaseJoiningPredicateBuilder im
List<Condition> andPredicates = new ArrayList<>();
List<List<IQueryParameterType>> chainParamValues = Collections.singletonList(orValues);
andPredicates.add(childQueryFactory.searchForIdsWithAndOr(
myColumnTargetResourceId,
subResourceName,
chain,
chainParamValues,
theRequest,
theRequestPartitionId,
SearchContainedModeEnum.FALSE));
andPredicates.add(
childQueryFactory.searchForIdsWithAndOr(with().setSourceJoinColumn(myColumnTargetResourceId)
.setResourceName(subResourceName)
.setParamName(chain)
.setAndOrParams(chainParamValues)
.setRequest(theRequest)
.setRequestPartitionId(theRequestPartitionId)));
orPredicates.add(QueryParameterUtils.toAndPredicate(andPredicates));
}

View File

@ -699,15 +699,24 @@ public class SearchQueryBuilder {
public ComboCondition addPredicateLastUpdated(DateRangeParam theDateRange) {
ResourceTablePredicateBuilder resourceTableRoot = getOrCreateResourceTablePredicateBuilder(false);
return addPredicateLastUpdated(theDateRange, resourceTableRoot);
}
public ComboCondition addPredicateLastUpdated(
DateRangeParam theDateRange, ResourceTablePredicateBuilder theResourceTablePredicateBuilder) {
List<Condition> conditions = new ArrayList<>(2);
BinaryCondition condition;
if (isNotEqualsComparator(theDateRange)) {
condition = createConditionForValueWithComparator(
LESSTHAN, resourceTableRoot.getLastUpdatedColumn(), theDateRange.getLowerBoundAsInstant());
LESSTHAN,
theResourceTablePredicateBuilder.getLastUpdatedColumn(),
theDateRange.getLowerBoundAsInstant());
conditions.add(condition);
condition = createConditionForValueWithComparator(
GREATERTHAN, resourceTableRoot.getLastUpdatedColumn(), theDateRange.getUpperBoundAsInstant());
GREATERTHAN,
theResourceTablePredicateBuilder.getLastUpdatedColumn(),
theDateRange.getUpperBoundAsInstant());
conditions.add(condition);
return ComboCondition.or(conditions.toArray(new Condition[0]));
}
@ -715,7 +724,7 @@ public class SearchQueryBuilder {
if (theDateRange.getLowerBoundAsInstant() != null) {
condition = createConditionForValueWithComparator(
GREATERTHAN_OR_EQUALS,
resourceTableRoot.getLastUpdatedColumn(),
theResourceTablePredicateBuilder.getLastUpdatedColumn(),
theDateRange.getLowerBoundAsInstant());
conditions.add(condition);
}
@ -723,7 +732,7 @@ public class SearchQueryBuilder {
if (theDateRange.getUpperBoundAsInstant() != null) {
condition = createConditionForValueWithComparator(
LESSTHAN_OR_EQUALS,
resourceTableRoot.getLastUpdatedColumn(),
theResourceTablePredicateBuilder.getLastUpdatedColumn(),
theDateRange.getUpperBoundAsInstant());
conditions.add(condition);
}

View File

@ -176,6 +176,7 @@ import static ca.uhn.fhir.rest.param.ParamPrefixEnum.LESSTHAN;
import static ca.uhn.fhir.rest.param.ParamPrefixEnum.LESSTHAN_OR_EQUALS;
import static ca.uhn.fhir.rest.param.ParamPrefixEnum.NOT_EQUAL;
import static ca.uhn.fhir.test.utilities.CustomMatchersUtil.assertDoesNotContainAnyOf;
import static ca.uhn.fhir.util.DateUtils.convertDateToIso8601String;
import static org.apache.commons.lang3.StringUtils.countMatches;
import static org.apache.commons.lang3.StringUtils.leftPad;
import static org.hamcrest.CoreMatchers.is;
@ -447,6 +448,57 @@ public class FhirResourceDaoR4SearchNoFtTest extends BaseJpaR4Test {
assertEquals(0, ids.size());
}
@Test
public void testHasEncounterAndLastUpdated() {
// setup
Patient patientA = new Patient();
String patientIdA = myPatientDao.create(patientA).getId().toUnqualifiedVersionless().getValue();
Patient patientB = new Patient();
String patientIdB = myPatientDao.create(patientA).getId().toUnqualifiedVersionless().getValue();
Encounter encounterA = new Encounter();
encounterA.getClass_().setSystem("http://snomed.info/sct").setCode("55822004");
encounterA.getSubject().setReference(patientIdA);
// record time between encounter A and B
TestUtil.sleepOneClick();
Date beforeA = new Date();
TestUtil.sleepOneClick();
myEncounterDao.create(encounterA);
Encounter encounterB = new Encounter();
encounterB.getClass_().setSystem("http://snomed.info/sct").setCode("55822005");
encounterB.getSubject().setReference(patientIdB);
// record time between encounter A and B
TestUtil.sleepOneClick();
Date beforeB = new Date();
TestUtil.sleepOneClick();
myEncounterDao.create(encounterB);
// execute
String criteriaA = "_has:Encounter:patient:_lastUpdated=ge" + convertDateToIso8601String(beforeA);
SearchParameterMap mapA = myMatchUrlService.translateMatchUrl(criteriaA, myFhirContext.getResourceDefinition(Patient.class));
mapA.setLoadSynchronous(true);
myCaptureQueriesListener.clear();
IBundleProvider resultA = myPatientDao.search(mapA);
myCaptureQueriesListener.logSelectQueries();
List<String> idsBeforeA = toUnqualifiedVersionlessIdValues(resultA);
String criteriaB = "_has:Encounter:patient:_lastUpdated=ge" + convertDateToIso8601String(beforeB);
SearchParameterMap mapB = myMatchUrlService.translateMatchUrl(criteriaB, myFhirContext.getResourceDefinition(Patient.class));
mapB.setLoadSynchronous(true);
IBundleProvider resultB = myPatientDao.search(mapB);
List<String> idsBeforeB = toUnqualifiedVersionlessIdValues(resultB);
// verify
assertEquals(2, idsBeforeA.size());
assertEquals(1, idsBeforeB.size());
}
@Test
public void testGenderBirthdateHasCondition() {
Patient patient = new Patient();