Difference between revisions of "Query Structures Web Service"

From FMR Knowledge Base
Jump to navigation Jump to search
(→‎Overview)
(→‎Headers)
 
(24 intermediate revisions by 3 users not shown)
Line 3: Line 3:
 
Fusion Metadata Registry provides a web service for '''querying''' structural metadata which conforms to the [[REST_API_Overview#Structural_Metadata_Queries|SDMX specification]].
 
Fusion Metadata Registry provides a web service for '''querying''' structural metadata which conforms to the [[REST_API_Overview#Structural_Metadata_Queries|SDMX specification]].
  
−
In addition to the SDMX specification, the Fusion Registry supports some additional Accept header values and
+
Some additional Accept header values and query parameters are also supported both of which are explained in this guide.
−
query parameters.
+
−
 
 
−
Both the SDMX and extended query parameters are included in this guide.
 
−
 
 
−
 
 
 
{| class="wikitable"
 
{| class="wikitable"
 
|-
 
|-
 
|style="background-color:#eaecf0"|<b>Entry Point</b>|| <b> /ws/public/sdmxapi/rest/ </b>
 
|style="background-color:#eaecf0"|<b>Entry Point</b>|| <b> /ws/public/sdmxapi/rest/ </b>
 
|-
 
|-
−
|style="background-color:#eaecf0"|<b>Access</b>|| <span style='color:green'><b>Secure</b></span> Public
+
|style="background-color:#eaecf0"|<b>Access</b>|| <span style='color:green'><b>Public</b></span>
 
|-
 
|-
 
|style="background-color:#eaecf0"|<b>Authentication</b>|| None
 
|style="background-color:#eaecf0"|<b>Authentication</b>|| None
Line 26: Line 22:
 
|-
 
|-
 
|style="background-color:#eaecf0"|<b>Error Response</b>|| SDMX-ML 2.1 Error Response Document
 
|style="background-color:#eaecf0"|<b>Error Response</b>|| SDMX-ML 2.1 Error Response Document
 +
|}
 +
 +
=Headers=
 +
{| class="wikitable"
 +
|-
 +
!| HTTP Header || Purpose || Allowed Values
 +
|-
 +
|style="background-color:#eaecf0; width:15%;"|<b>Accept</b>|| style="width:42%;" |To define the response format || 
 +
<ul>
 +
<li>application/vnd.sdmx.structure+edi</li>
 +
<li>application/vnd.sdmx.structure+xml;version=1.0</li>
 +
<li>application/vnd.sdmx.structure+xml;version=2.0</li>
 +
<li>application/vnd.sdmx.structure+xml;version=2.1</li>
 +
<li>application/vnd.sdmx.json</li>
 +
</ul>
 +
|-
 +
|style="background-color:#eaecf0"|<b>Accept-Language</b>||
 +
This optional header can be used to set the locale to return any multilingual text in (names and descriptions). If the text does not exist in the specified locale, then the default rules will be applied
 +
to find the next best appropriate locale.
 +
|
 +
Examples:<br>
 +
Accept-Language : en (English)<br>
 +
Accept-Language : fr (French)<br>
 +
Accept-Language : * (all languages – no filter)<br>
 +
Accept-Language : all (all languages – no filter)<br>
 +
|-
 +
|style="background-color:#eaecf0; width:15%;"|<b>If-Modified-Since</b>|| style="width:42%;" | The server sends back the requested resource, with a 200 status, only if it has been last modified after the specified date. If the resource has not been modified since, the response is a 304 without any body. <br>
 +
'''Note:''' The current implementation has a limitiation in that any deletion of structural metadata will reset all datetimes of existing structures to the current time. This is to avoid the "false negative" scenario whereby querying for all structures would not return a 200 response if a structure had been deleted. The current implementation does give "false positives" - i.e. a structure has changed when it has not.
 +
|| A date in the expected format of: <br> <Day-Name>, <Day> <Month-Name> <Year> <Hour>:<Minute>:<Second> GMT <br> Examples: <br>
 +
Fri, 31 Dec 2023 23:59:59 GMT<br>
 +
Thu, 8 June 2023 14:00:00 GMT
 +
 +
|}
 +
 +
=Resources=
 +
The following resources are defined:
 +
 +
* datastructure (This has been shortened from DataStructureDefinition to allow for shorter URLs)
 +
* metadatastructure (This has been shortened from MetadataStructureDefinition to allow for shorter URLs)
 +
* categoryscheme
 +
* conceptscheme
 +
* codelist
 +
* hierarchicalcodelist
 +
* organisationscheme (The organisationscheme resource can be used whenever the role played by the organisation schemes is not known/relevant)
 +
* agencyscheme (For 3 of the subtypes of OrganisationScheme, the id and version parameters have fixed values. See Section 03 of the SDMX information model document for additional information)
 +
* dataproviderscheme
 +
* dataconsumerscheme
 +
* organisationunitscheme
 +
* dataflow
 +
* metadataflow
 +
* reportingtaxonomy
 +
* provisionagreement
 +
* structureset
 +
* process
 +
* categorisation
 +
* contentconstraint
 +
* actualconstraint (a type of contentconstraint stating what data are actually present)
 +
* allowedconstraint (a type of contentconstraint defining what data are allowed)
 +
* attachmentconstraint
 +
* transformationscheme
 +
* rulesetscheme
 +
* userdefinedoperatorscheme
 +
* customtypescheme
 +
* namepersonalisationscheme
 +
* vtlmappingscheme
 +
* structure (This type can be used to retrieve any type of structural metadata matching the supplied parameters)
 +
 +
=Parameters=
 +
The following parameters are used for identifying resources:
 +
 +
{| class="wikitable"
 +
! Parameter !! Type !! Description
 +
|-
 +
| agencyID
 +
| A string compliant with the SDMX ''common:NCNameIDType''
 +
| The agency maintaining the artefact to be returned. It is possible to set more than one agency, using <code>+</code> as separator (e.g. BIS+ECB).
 +
|-
 +
| resourceID
 +
| A string compliant with the SDMX ''common:IDType''
 +
| The id of the artefact to be returned. It is possible to set more than one id, using <code>+</code> as separator (e.g. CL_FREQ+CL_CONF_STATUS).
 +
|-
 +
| version
 +
| A string compliant with the SDMX ''common:VersionType''
 +
| The version of the artefact to be returned. It is possible to set more than one version, using <code>+</code> as separator (e.g. 1.0+2.1).
 +
|}
 +
 +
The parameters mentioned above are specified using the following syntax:
 +
 +
<pre>protocol://domain/ws/public/sdmxapi/rest/resource/agencyID/resourceID/version</pre>
 +
 +
= Additional parameter used for identifying a resource, for item scheme types =
 +
 +
SDMX uses the ''item scheme'' pattern to model SDMX collections of items. These are principally:
 +
 +
* categoryscheme
 +
* conceptscheme
 +
* codelist
 +
... and others concerning organisation schemes and VTL transformations.<br>
 +
A complete list can be found in the [[REST_API_Overview#Additional_parameter_used_for_identifying_a_resource.2C_for_item_scheme_types|REST API Overview]].
 +
 +
Although it is not following the ''item scheme'' pattern, ''hierarchicalcodelist'' is also a collection, i.e. a collection of hierarchies.
 +
 +
For these collections (those following the ''item scheme'' pattern or the ''hierarchicalcodelist''), it is possible to use a 4th parameter for identifying a resource. The rules for the 3 other parameters, as defined in the section above, remain valid.
 +
 +
{| class="wikitable"
 +
! Parameter !! Type !! Description
 +
|-
 +
| itemID
 +
| A string compliant with the SDMX ''common:NestedNCNameIDType'' for conceptscheme and agencyscheme, SDMX ''common:IDType'' for hierarchicalcodelist or with the SDMX ''common:NestedIDType'' in all other cases
 +
| The id of the item to be returned. It is possible to set more than one id, using <code>+</code> as separator (e.g. A+Q+M).
 +
|}
 +
 +
This 4th parameter is used as follows:
 +
 +
<pre>protocol://domain/ws/public/sdmxapi/rest/resource/agencyID/resourceID/version/itemID</pre>
 +
 +
=URL Request Parameters=
 +
The request parameters are all optional and can be used to define the response detail, format, and any
 +
additional structures which reference, or are referenced by those identified in the query path.
 +
{| class="wikitable"
 +
! '''Parameter'''
 +
! '''Purpose'''
 +
! '''Allowed Values'''
 +
|-
 +
| '''detail'''
 +
| To define which structures (if any) are output as stubs.
 +
|
 +
'''full''' – default. Output full response.
 +
 +
'''allstubs –''' Output all the structures as stubs.
 +
 +
'''referencestubs''' – Output the full query result, and any referenced structures are returned as stubs
 +
 +
'''referencepartial''' – Outputs the full query result and any referenced Codelists, Concept Schemes, Agency Schemes are returned as partial lists based on the Codes, Concepts, and Agencies used by the referencing Provision Agreements, Dataflows, Data Structures, Hierarchical Codelists. Partial Codelists are derived from Content Constraints used to define allowable content for data reporting.
 +
 +
''Example:''
 +
 +
detail=allstubs
 +
 +
detail=referencepartial
 +
|-
 +
| '''references'''
 +
|
 +
To define if additional structures are returned from the query.
 +
 +
The structures can either be ones which reference, or are referenced by the structures in the query result.
 +
 +
If the query result is for a specific item in an item scheme, then this parameter will identify the references for that item.
 +
|
 +
'''none''' – default. Do not output any additional structures
 +
 +
'''parents''' –output structures the reference the structures matching the query
 +
 +
'''parentsandsiblings''' – same as parents, but also include all the additional structures referenced by the parents
 +
 +
'''children''' – the structures referenced by the structures in the query result
 +
 +
'''descendants''' – children and their children (up to any level)
 +
 +
In addition, a concrete type of resource may be used, for example:
 +
 +
'''datastructure '''
 +
 +
''Example:''
 +
 +
references=datastructure
 +
|-
 +
| '''partial'''
 +
|
 +
If set to true creates partial Codelists in the response based on the Fusion Metadata Registry Content Constraints defining allowable content.
 +
 +
The pre-requisite is that the query must be for a single constrainable structure (Provision Agreements, Dataflow, or Data Structure) and include references.
 +
|
 +
true/false
 +
 +
'''Note:''' This is deprecated as of v9.2.23, use detail=referencepartial instead
 +
|-
 +
| '''format'''
 +
| Can be used to define the response format (as an alternative to the HTTP Accept Header).
 +
|
 +
sdmx (latest version)
 +
 +
sdmx-1.0
 +
 +
sdmx-2.0
 +
 +
sdmx-2.1
 +
 +
sdmx-edi
 +
 +
sdmx-edi-lenient
 +
 +
sdmx-json
 +
 +
rdf-turtle
 +
 +
rdf-json
 +
 +
rdf-xml
 +
 +
xlsx
 +
 +
''Example:''
 +
 +
format=sdmx-edi
 +
|-
 +
| '''includeMetadata'''
 +
|
 +
If set to true, then the response structures will contain additional Annotations if there are reference metadata attached.
 +
 +
The annotation will have the AnnotationType of ‘METADATA’ and the AnnotationURI will provide a URl to the MetadataSet.
 +
|
 +
true/false
 +
 +
''Example:''
 +
 +
includeMetadata=true
 +
|-
 +
| '''locale'''
 +
|
 +
This optional parameter can be used to set the locale to return any multilingual text in (names and descriptions). If the text does not exist in the specified locale, then the default rules will be applied to find the next best appropriate locale.
 +
 +
This takes priority over the Accept-Language HTTP header
 +
|
 +
Any locale
 +
 +
''Example:''
 +
 +
locale=fr
 +
|-
 +
| '''includeMetrics'''
 +
| Enriches structures with information about data that is available. See Annex 3 – Metrics and Data Availability
 +
|
 +
true/false/only
 +
 +
IncludeMetrics=only is used to indicate that only maintainable structures with metrics should be returned, and maintainables which have no metrics are removed from the query response.
 +
 +
''Example:''
 +
 +
includeMetrics=true
 +
 +
includeMetrics=only
 +
|-
 +
| '''saveAs'''
 +
|
 +
If provided the HTTP Header ‘Content-Disposition’ will be set to attachment with the filename being set to the value provided.
 +
 +
This will result in the response being saved to a file.
 +
 +
The file extension is not required as the Registry will determine the extension based on the response format.
 +
|
 +
String
 +
 +
''Example:''
 +
 +
saveAs=myDownload
 +
|-
 +
| '''prettyPrint'''
 +
| If the you are requesting XML, and you would like the response XML to be formatted, then you can pass true
 +
|
 +
String
 +
 +
prettyPrint=true
 +
|-
 +
| '''validFrom'''
 +
| For structures with a defined validFrom or validTo value, returns only those structures which have a validFrom value before the specified date
 +
|
 +
Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used
 +
 +
''Example:''
 +
 +
validFrom=1960-12-31
 +
|-
 +
| '''validTo'''
 +
| For structures with a defined validFrom or validTo value, returns only those structures which have a validTo value after the specified date
 +
|
 +
Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used
 +
 +
''Example:''
 +
 +
validTo=1960-12-31
 +
|-
 +
| '''validOn'''
 +
| Returns structures where the items listed are applicable for the specified date. This parameter is only applicable for those structures which support Item Validity.
 +
|
 +
Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used
 +
 +
''Example:''
 +
 +
validOn=1960-12-31
 +
|-
 +
| '''labels'''
 +
|
 +
'''For sdmx-csv format only'''
 +
 +
For each element of the CSV, will return either the ID only or the ID and Name separated by a colon
 +
|
 +
Id – default
 +
 +
both
 +
 +
''Example:''
 +
 +
labels=both
 +
|-
 +
| '''timeFormat'''
 +
|
 +
'''For sdmx-csv format only'''
 +
 +
Normalized TIME_PERIOD values are converted to the most granular ISO 8601 representation taking into account the highest frequency of the data in the message
 +
|
 +
original – default
 +
 +
normalized
 +
 +
''Example:''
 +
 +
timeFormat=normalized
 
|}
 
|}

Latest revision as of 02:19, 9 June 2023

Overview

Fusion Metadata Registry provides a web service for querying structural metadata which conforms to the SDMX specification.

Some additional Accept header values and query parameters are also supported both of which are explained in this guide.

Entry Point /ws/public/sdmxapi/rest/
Access Public
Authentication None
Http Method GET
Response Format Multiple SDMX formats supported which can be specified wither using the HTTP Accept header or the 'format' URL parameter
Response Statuses

200 - Success

404 - Structure not found

500 - Server Error

Error Response SDMX-ML 2.1 Error Response Document

Headers

HTTP Header Purpose Allowed Values
Accept To define the response format
  • application/vnd.sdmx.structure+edi
  • application/vnd.sdmx.structure+xml;version=1.0
  • application/vnd.sdmx.structure+xml;version=2.0
  • application/vnd.sdmx.structure+xml;version=2.1
  • application/vnd.sdmx.json
Accept-Language

This optional header can be used to set the locale to return any multilingual text in (names and descriptions). If the text does not exist in the specified locale, then the default rules will be applied to find the next best appropriate locale.

Examples:
Accept-Language : en (English)
Accept-Language : fr (French)
Accept-Language : * (all languages – no filter)
Accept-Language : all (all languages – no filter)

If-Modified-Since The server sends back the requested resource, with a 200 status, only if it has been last modified after the specified date. If the resource has not been modified since, the response is a 304 without any body.

Note: The current implementation has a limitiation in that any deletion of structural metadata will reset all datetimes of existing structures to the current time. This is to avoid the "false negative" scenario whereby querying for all structures would not return a 200 response if a structure had been deleted. The current implementation does give "false positives" - i.e. a structure has changed when it has not.

A date in the expected format of:
<Day-Name>, <Day> <Month-Name> <Year> <Hour>:<Minute>:<Second> GMT
Examples:

Fri, 31 Dec 2023 23:59:59 GMT
Thu, 8 June 2023 14:00:00 GMT

Resources

The following resources are defined:

  • datastructure (This has been shortened from DataStructureDefinition to allow for shorter URLs)
  • metadatastructure (This has been shortened from MetadataStructureDefinition to allow for shorter URLs)
  • categoryscheme
  • conceptscheme
  • codelist
  • hierarchicalcodelist
  • organisationscheme (The organisationscheme resource can be used whenever the role played by the organisation schemes is not known/relevant)
  • agencyscheme (For 3 of the subtypes of OrganisationScheme, the id and version parameters have fixed values. See Section 03 of the SDMX information model document for additional information)
  • dataproviderscheme
  • dataconsumerscheme
  • organisationunitscheme
  • dataflow
  • metadataflow
  • reportingtaxonomy
  • provisionagreement
  • structureset
  • process
  • categorisation
  • contentconstraint
  • actualconstraint (a type of contentconstraint stating what data are actually present)
  • allowedconstraint (a type of contentconstraint defining what data are allowed)
  • attachmentconstraint
  • transformationscheme
  • rulesetscheme
  • userdefinedoperatorscheme
  • customtypescheme
  • namepersonalisationscheme
  • vtlmappingscheme
  • structure (This type can be used to retrieve any type of structural metadata matching the supplied parameters)

Parameters

The following parameters are used for identifying resources:

Parameter Type Description
agencyID A string compliant with the SDMX common:NCNameIDType The agency maintaining the artefact to be returned. It is possible to set more than one agency, using + as separator (e.g. BIS+ECB).
resourceID A string compliant with the SDMX common:IDType The id of the artefact to be returned. It is possible to set more than one id, using + as separator (e.g. CL_FREQ+CL_CONF_STATUS).
version A string compliant with the SDMX common:VersionType The version of the artefact to be returned. It is possible to set more than one version, using + as separator (e.g. 1.0+2.1).

The parameters mentioned above are specified using the following syntax: 

protocol://domain/ws/public/sdmxapi/rest/resource/agencyID/resourceID/version

Additional parameter used for identifying a resource, for item scheme types

SDMX uses the item scheme pattern to model SDMX collections of items. These are principally:

  • categoryscheme
  • conceptscheme
  • codelist

... and others concerning organisation schemes and VTL transformations.
A complete list can be found in the REST API Overview.

Although it is not following the item scheme pattern, hierarchicalcodelist is also a collection, i.e. a collection of hierarchies.

For these collections (those following the item scheme pattern or the hierarchicalcodelist), it is possible to use a 4th parameter for identifying a resource. The rules for the 3 other parameters, as defined in the section above, remain valid.

Parameter Type Description
itemID A string compliant with the SDMX common:NestedNCNameIDType for conceptscheme and agencyscheme, SDMX common:IDType for hierarchicalcodelist or with the SDMX common:NestedIDType in all other cases The id of the item to be returned. It is possible to set more than one id, using + as separator (e.g. A+Q+M).

This 4th parameter is used as follows: 

protocol://domain/ws/public/sdmxapi/rest/resource/agencyID/resourceID/version/itemID

URL Request Parameters

The request parameters are all optional and can be used to define the response detail, format, and any additional structures which reference, or are referenced by those identified in the query path.

Parameter Purpose Allowed Values
detail To define which structures (if any) are output as stubs.

full – default. Output full response.

allstubs – Output all the structures as stubs.

referencestubs – Output the full query result, and any referenced structures are returned as stubs

referencepartial – Outputs the full query result and any referenced Codelists, Concept Schemes, Agency Schemes are returned as partial lists based on the Codes, Concepts, and Agencies used by the referencing Provision Agreements, Dataflows, Data Structures, Hierarchical Codelists. Partial Codelists are derived from Content Constraints used to define allowable content for data reporting.

Example:

detail=allstubs

detail=referencepartial

references

To define if additional structures are returned from the query.

The structures can either be ones which reference, or are referenced by the structures in the query result.

If the query result is for a specific item in an item scheme, then this parameter will identify the references for that item.

none – default. Do not output any additional structures

parents –output structures the reference the structures matching the query

parentsandsiblings – same as parents, but also include all the additional structures referenced by the parents

children – the structures referenced by the structures in the query result

descendants – children and their children (up to any level)

In addition, a concrete type of resource may be used, for example:

datastructure

Example:

references=datastructure

partial

If set to true creates partial Codelists in the response based on the Fusion Metadata Registry Content Constraints defining allowable content.

The pre-requisite is that the query must be for a single constrainable structure (Provision Agreements, Dataflow, or Data Structure) and include references.

true/false

Note: This is deprecated as of v9.2.23, use detail=referencepartial instead

format Can be used to define the response format (as an alternative to the HTTP Accept Header).

sdmx (latest version)

sdmx-1.0

sdmx-2.0

sdmx-2.1

sdmx-edi

sdmx-edi-lenient

sdmx-json

rdf-turtle

rdf-json

rdf-xml

xlsx

Example:

format=sdmx-edi

includeMetadata

If set to true, then the response structures will contain additional Annotations if there are reference metadata attached.

The annotation will have the AnnotationType of ‘METADATA’ and the AnnotationURI will provide a URl to the MetadataSet.

true/false

Example:

includeMetadata=true

locale

This optional parameter can be used to set the locale to return any multilingual text in (names and descriptions). If the text does not exist in the specified locale, then the default rules will be applied to find the next best appropriate locale.

This takes priority over the Accept-Language HTTP header

Any locale

Example:

locale=fr

includeMetrics Enriches structures with information about data that is available. See Annex 3 – Metrics and Data Availability

true/false/only

IncludeMetrics=only is used to indicate that only maintainable structures with metrics should be returned, and maintainables which have no metrics are removed from the query response.

Example:

includeMetrics=true

includeMetrics=only

saveAs

If provided the HTTP Header ‘Content-Disposition’ will be set to attachment with the filename being set to the value provided.

This will result in the response being saved to a file.

The file extension is not required as the Registry will determine the extension based on the response format.

String

Example:

saveAs=myDownload

prettyPrint If the you are requesting XML, and you would like the response XML to be formatted, then you can pass true

String

prettyPrint=true

validFrom For structures with a defined validFrom or validTo value, returns only those structures which have a validFrom value before the specified date

Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used

Example:

validFrom=1960-12-31

validTo For structures with a defined validFrom or validTo value, returns only those structures which have a validTo value after the specified date

Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used

Example:

validTo=1960-12-31

validOn Returns structures where the items listed are applicable for the specified date. This parameter is only applicable for those structures which support Item Validity.

Single time period. Conforms to the ISO-8601 standard but SDMX date time formats may also be used

Example:

validOn=1960-12-31

labels

For sdmx-csv format only

For each element of the CSV, will return either the ID only or the ID and Name separated by a colon

Id – default

both

Example:

labels=both

timeFormat

For sdmx-csv format only

Normalized TIME_PERIOD values are converted to the most granular ISO 8601 representation taking into account the highest frequency of the data in the message

original – default

normalized

Example:

timeFormat=normalized

Retrieved from "https://fmrwiki.sdmxcloud.org/index.php?title=Query_Structures_Web_Service&oldid=6906"