USRealEstate
/
web-api-commander
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
web-api-commander/doc/Certification.md

18 KiB
Raw Blame History

RESO Certification

The RESO Commander is the basis for automated Data Dictionary, Payloads, and Web API Certification.

Java and the JDK

To run the Commander as an automated testing tool, the Java JDK must be installed. The Commander has been tested with JDK 1.8 and 10 at this point. Those using JDK 11+, please report issues if they arise.

To see whether you have the JDK installed, type the following using your local command line environment:

$ java -version

If you have a Java JDK installed, your output will look something like:

$ java -version
openjdk version "1.8.0_275"
OpenJDK Runtime Environment (build 1.8.0_275-8u275-b01-0ubuntu1~20.10-b01)
OpenJDK 64-Bit Server VM (build 25.275-b01, mixed mode)

If you don't see something like this, you need to install the JDK:

Note: there are known issues with Java/JDK 14+ and Groovy. The recommended JDK versions are 8 (1.8), 10, or 11. OpenJDK is preferred. If you're using MacOS you can install the JDK using Homebrew. There are instructions for Windows and Linux here.

Cloning Commander Repository

The Commander may be run in automated testing mode using a terminal. Automated testing assumes that you have a Java 1.8+ JDK installed, as mentioned elsewhere in this README.

First, change into the directory you want to work in and clone the Commander repository.

You will need to have Git installed. Chances are you already do, to check, open a command line and type git and if it's present, it will print some info about the app. If not, there are installation instructions here.

MacOS or Linux
$ git clone https://github.com/RESOStandards/web-api-commander.git
Windows
C:\> git clone https://github.com/RESOStandards/web-api-commander.git

This will clone the repository into a directory called web-api-commander relative to whatever directory you're currently in, which also means you'll have a fresh copy of the latest code to execute.

To refresh the code after you have downloaded it, issue the command $ git pull in the root of the directory that was just created.

Cucumber Feature Specifications

Cucumber is being used to describe acceptance criteria in a higher-level DSL rather than encapsulating all of the test logic code. Cucumber's DSL is called Gherkin and essentially allows backing test code to be organized in a logical manner that makes sense to analysts as well as programmers.

Testing Environment

Under the hood, Gradle is being used for automation. It works across multiple platforms and is friendly with both Docker and Cucumber so that tests may be automated on CI/CD platforms such as Jenkins, Circle CI, Travis, or similar, and emit standard system codes during regression testing.

Gradle Wrapper

The Gradle wrapper provides a convenient way to automatically download Gradle when running tests.

After you have cloned the repository, as shown in a previous step, change into the directory containing the source code from GitHub. Convenience methods have been provided for the various certification tasks.

Gradle Tasks

Once the Gradle Wrapper is set up, you should be able to run the ./gradlew tasks command in from the root of the Commander source directory in a terminal window and see the list of available tasks.

$ ./gradlew tasks

> Task :tasks

------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
...

There are both built-in tasks and RESO tasks.

The following section is what's of interest here:

RESO Certification tasks
------------------------
testDataAvailability_1_7 - Data Dictionary 1.7 Data Availability Tests
Example:
  ./gradlew testDataAvailability_1_7 -DpathToRESOScript=/path/to/web-api-core-1.0.2.resoscript
[Report location: build/certification/reports]


testDataDictionary_1_7 - Data Dictionary 1.7 Acceptance Tests
RESOScript Example:
  ./gradlew testDataDictionary_1_7 -DpathToRESOScript=/path/to/dd17.resoscript -DshowResponses=true -Dstrict=true

Metadata File Example:
  ./gradlew testDataDictionary_1_7 -DpathToMetadata=/path/to/RESODataDictionary-1.7.xml

To disable strict mode, remove the -Dstrict=true parameter. All applicants MUST pass strict mode tests to be certified!

[Report location: build/certification/reports]


testIdxPayload_1_7 - Data Dictionary 1.7 Payloads Sampling Tests
Example:
  ./gradlew testIdxPayload_1_7 -DpathToRESOScript=/path/to/web-api-core-1.0.2.resoscript

[Report location: build/certification/reports]


testWebApiCore_1_0_2 - Web API Core 1.0.2 Acceptance Tests
Example: 
  ./gradlew testWebApiCore_1_0_2 -DpathToRESOScript=/path/to/web-api-core-1.0.2.resoscript -DshowResponses=true

Note: by default the Web API tests assume Collection(Edm.EnumType).
Pass -DuseCollections=false if using OData IsFlags.

[Report location: build/certification/reports]

Automated RESO Web API Core Testing

Automated Web API Core automated testing tools are currently in development. See Issue 34 for progress.

To use the automated RESO testing tools, you must have a JDK installed.

Web API Core RESOScript Template

To use the Commander for automated Web API Core testing, you need a RESOScript.

For Web API 1.0.2 Server Core Certification, use this resoscript as a template.

For more information regarding Parameters and Client Settings, see the Web API Walkthrough (in-progress).

Web API Cucumber Acceptance Tests

The Cucumber BDD acceptance tests for Web API 1.0.2 Core certification are here. If you have any questions, please send us an email.

Gradle Tasks for Web API 1.0.2 Server Certification

While you may use tags to filter tests as you choose, explained in the next section, it's convenient to be able to run a predefined set of tests Web API Core Certification.

These tasks will also produce reports in the local build directory, named according to which test you ran.

Core Certification

This will run the Core tests against the Web API 1.0.2 Server provided as WebAPIURI in your web-api-server.core.1.0.2.resoscript file.

Note: by default, the Commander uses Collection(Edm.EnumType) for multiple enumerations testing. Pass -DuseCollections=false if you are using IsFlags="true" instead.

MacOS or Linux
$ ./gradlew testWebApiCore_1_0_2 -DpathToRESOScript=/path/to/your.web-api-server.core.1.0.2.resoscript -DshowResponses=true
Windows
C:\path\to\web-api-commander> gradlew testWebApiCore_1_0_2 -DpathToRESOScript=C:\path\to\your.web-api-server.core.1.0.2.resoscript -DshowResponses=true

*Note: the first time you run these tasks, they will take some time as the environment must be configured and code is being compiled from the contents of the source directory downloaded in the previous step.

Web API Program Output

A sample of the runtime terminal output follows:

> Task :testWebApiCore_1_0_2

@metadata-request @2.4.1
Scenario: REQ-WA103-END3 - Request and Validate Server Metadata                                            

Using RESOScript: ./web-api-server.core.1.0.2.resoscript
  Given a RESOScript file was provided                                                                     

RESOScript loaded successfully!
  And Client Settings and Parameters were read from the file                                               

Bearer token loaded... first 4 characters: test
Service root is: https://api.yourserver.com/OData
  And a test container was successfully created from the given RESOScript                                  

Authentication Type: authorization_code
  And the test container uses an authorization_code or client_credentials for authentication               

Requesting XML Metadata from service root at: https://api.yourserver.com/OData
  When XML Metadata are requested from the service root in "ClientSettings_WebAPIURI"                      

Asserted Response Code: 200, Server Response Code: 200
  Then the server responds with a status code of 200                                                       

Reported OData-Version header value: '4.0'
  And the server has an OData-Version header value of "4.0" or "4.01"                                      

Validating XML Metadata response to ensure it's valid XML and matches OASIS OData XSDs...
See: https://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/schemas/
XMLMetadata string is valid XML!
  And the XML Metadata response is valid XML                                                               

Validating XML Metadata response to ensure it's valid XML and matches OASIS OData XSDs...
See: https://docs.oasis-open.org/odata/odata/v4.0/errata03/os/complete/schemas/
XMLMetadata string is valid XML!
XML Metadata is valid!
Edm Metadata is valid!
  And the XML Metadata returned by the server are valid                                                    

  And the XML Metadata returned by the server contains Edm metadata                                        

  And the Edm metadata returned by the server are valid                                                    

Found Default Entity Container: 'Default'
  And the metadata contains a valid service document                                                       

Resource Name: Property
Allowed Resources: Property, Member, Office, Contacts, ContactListings, HistoryTransactional, InternetTracking, Media, OpenHouse, OUID, Prospecting, Queue, Rules, SavedSearch, Showing, Teams
  And the given "Parameter_EndpointResource" resource exists within "Parameter_DD17_WellKnownResourceList" 

Found EntityContainer for the given resource: 'Property'
  And the metadata contains the "Parameter_EndpointResource" resource                                      

Searching the default entity container for one of the following Standard Resources: Property, Member, Office, Media
Standard Resource Names requirement met!
  And the metadata contains at least one resource from "Parameter_WebAPI102_RequiredResourceList"          


1 Scenarios (1 passed)
15 Steps (15 passed)
0m4.093s

Detailed information will be added to a local commander.log file at runtime.

Automated RESO Data Dictionary Testing

The Commander provides automated Data Dictionary 1.7 acceptance testing for RESO Certification. The DD 1.7 testing specification is available here.

To use the RESO Commander for Data Dictionary testing, you must have the JDK installed and a local copy of the Commander repository. See RESO Certification before proceeding.

Data Dictionary RESOScript Template

To use the Commander for automated Data Dictionary testing, you need a RESOScript.

For Data Dictionary 1.7 Certification, use this resoscript as a template.

Data Dictionary Acceptance Tests

RESO Data Dictionary Certification is driven off of the official Data Dictionary spreadsheet for each version of the dictionary, currently DD 1.7.

Cucumber BDD acceptance tests are automatically generated from the local copy of the approved spreadsheet.

The generated Data Dictionary 1.7 Cucumber BDD tests are located in this directory. See the property.feature file, for example, for the RESO Property Resource acceptance tests.

If you have any questions, please send us an email.

Gradle Tasks for Data Dictionary Certification

There are predefined tasks for automated RESO Data Dictionary Certification using the Commander. These can be displayed using Gradle Tasks as well.

Note: the first time you run these tasks, they will take some time as the environment must be configured and code is being compiled from the contents of the source directory downloaded in the previous step.

Test Data Dictionary

This task tests for Data Dictionary compliance and generates a raw report in a timestamped local directory.

There are two ways to run automated testing to check for RESO compliant Web API metadata:

  • using a local metadata file
  • using a RESOScript file to fetch metadata from a given server

While RESOScript files and the use of strict mode are required for RESO Certification. In both cases, metadata are validated and then processed for RESO compliance.

Data Dictionary Testing using Local Metadata

The Commander allows for a local metadata file to be specified. Not only is this used for internal acceptance testing, but is useful for developers to troubleshoot metadata locally while working on compliance.

The Gradle task to validate local metadata can be run using the following command:

$ ./gradlew testDataDictionary_1_7 -DpathToMetadata=/path/to/RESODataDictionary-1.7.xml

You may also pass a -Dstrict=true flag to see whether the given metadata file would pass Certification.

A raw report will be generated in a timestamped directory, and a commander.log will be generated during runtime.

Data Dictionary Testing using a Data Dictionary RESOScript

During Certification, metadata are retrieved directly from an applicant's Web API server using either OAuth2 Bearer Tokens or Client Credentials. Either authentication option is currently available for RESO Certification, depending on configuration, and the applicant will provide working RESOScripts when they apply for certification.

An example Data Dictionary RESOScript template can be found here.

Once a RESOScript file has been created, it may be used with the following command:

$ ./gradlew testDataDictionary_1_7 -DpathToRESOScript=/path/to/dd17.resoscript -DshowResponses=true

You may also pass a -Dstrict=true flag to see whether the given metadata file would pass Certification.

A raw report will be generated in a timestamped directory, and a commander.log will be generated during runtime.

Data Dictionary Testing Output

To see examples of Data Dictionary testing output, you may use the ./gradlew testDataDictionaryReferenceMetadata_1_7 command to run the Data Dictionary acceptance tests on the RESO reference metadata.

There is additional documentation about how Data Dictionary testing works, including sample output, in the RESO Data Dictionary 1.7 Testing Specification.

Advanced feature: Tag Filtering

You may filter by tags in any of the Web API or Data Dictionary tests. These are the items in the Cucumber .feature files prefixed by an @ symbol. Expressions may also be used with tags. This README doen't cover how to use tags, but the Commander supports them. For more information, see the Cucumber Documentation.

Examples

Run Web API Core Metadata Tests Only

$ gradle testWebApiCore_1_0_2 -DpathToRESOScript=/path/to/your.web-api-server.core.1.0.2.resoscript -Dcucumber.filter.tags="@metadata"

Run Data Dictionary Tests on IDX Fields Only

$ ./gradlew testDataDictionary_1_7 -DpathToRESOScript=/path/to/your/dd1.7.resoscript -DshowResponses=true -Dcucumber.filter.tags="@IDX"