Difference between revisions of "REST API Overview"

From FMR Knowledge Base
Jump to navigation Jump to search
(→‎Parameters used to further filter the desired results)
(→‎Authentication)
 
(70 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
[[Category:FMR_REST_API_Reference]]
 
[[Category:FMR_REST_API_Reference]]
 +
=SDMX Standard REST API Specification=
  
−
== Scope of the API ==
+
== Scope of the SDMX REST API ==
  
 
The RESTful API focuses on simplicity. The aim is not to replicate the full semantic richness of the SDMX-ML Query message but to make it simple to perform a limited set of standard queries. Also, in contrast to other parts of the SDMX specification, the RESTful API focuses solely on data retrieval (via HTTP GET). More specifically, the API allows:
 
The RESTful API focuses on simplicity. The aim is not to replicate the full semantic richness of the SDMX-ML Query message but to make it simple to perform a limited set of standard queries. Also, in contrast to other parts of the SDMX specification, the RESTful API focuses solely on data retrieval (via HTTP GET). More specifically, the API allows:
Line 14: Line 15:
 
* A distinction should be established between the elements that allow identifying the resource to be retrieved and the elements that give additional information about, or allow to further filter, the desired results. Elements belonging to the 1st category are specified in the path part of the URL while elements belonging to the 2nd category are specified in the query string part of the URL.
 
* A distinction should be established between the elements that allow identifying the resource to be retrieved and the elements that give additional information about, or allow to further filter, the desired results. Elements belonging to the 1st category are specified in the path part of the URL while elements belonging to the 2nd category are specified in the query string part of the URL.
  
−
== Structural Metadata Queries ==
+
==Implementation of the SDMX REST API in Fusion Metadata Registry==
 +
Fusion Metadata Registry is primarily concerned with structural metadata so implements a subset of the SDMX RESTful API specification.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! SDMX REST API Function !! Implemented
 +
|-
 +
| Structural Metadata Queries || style="backgroundcolor:red;" | YES
 +
|-
 +
| Schema Queries (for retrieving SDMX-ML Structure Specific Data validation schemas) || YES
 +
|-
 +
| Data Queries || NO
 +
|-
 +
| Data Availability || NO
 +
|}
 +
 
 +
The Fusion Metadata Registry API also provides some additional functions that are necessary for a practical registry implementation, but not explicitly provided for in SDMX standard REST specification.
 +
 
 +
{| class="wikitable"
 +
|-
 +
! FMR extended REST API Function !! Purpose
 +
|-
 +
| Structure Submission || The addition or modification of structures using HTTP POST
 +
|-
 +
| Structure Deletion || Delete specific structures using HTTP DELETE
 +
|-
 +
| Data Validation (synchronous) || Validates a dataset submitted using HTTP POST returning a validation report in JSON
 +
|-
 +
| Data Transformation (synchronous) || Transforms a dataset submitted using HTTP POST to a different DSD with an optional report returned in JSON
 +
|-
 +
| Data Transformation and Validation (asynchronous) || Validates and transforms datasets asynchronously making it suitable for larger submissions
 +
|-
 +
| Product Info || Returns details about the running Fusion Metadata Registry Instance in JSON
 +
|-
 +
| Active User || Returns details about the current authenticated user in JSON
 +
|}
 +
 
 +
= Structural Metadata Queries =
 +
 
 +
A structural metadata query uses HTTP GET with the following general URL syntax:
 +
<pre>http(s)://ws-entry-point/resource/agencyID/resourceID/version</pre>
  
 
=== Resources ===
 
=== Resources ===
Line 78: Line 119:
  
 
{| class="wikitable"
 
{| class="wikitable"
−
!width="5%"| Keywords
+
!Keywords !! Scope !! Description
−
!width="5%"| Scope
 
−
!width="5%"| Description
 
 
|-
 
|-
 
| all
 
| all
Line 104: Line 143:
  
 
* If no version is specified, the version currently used in production should be returned. It is therefore equivalent to using the keyword <code>latest</code>.
 
* If no version is specified, the version currently used in production should be returned. It is therefore equivalent to using the keyword <code>latest</code>.
−
* If no agencyID is specified, the matching artefacts maintained by any maintenance agency should be returned. It is therefore equivalent to using the keyword <code>all</code>. This would potentially return more than one artefact, if different agencies give the same identifier to a resource (for example, http://ws-entry-point/codelist/all/CL_FREQ, could return more than one codelist if more than one agency is maintaining a codelist with id &quot;CL_FREQ&quot;).
+
* If no agencyID is specified, the matching artefacts maintained by any maintenance agency should be returned. It is therefore equivalent to using the keyword <code>all</code>. This would potentially return more than one artefact, if different agencies give the same identifier to a resource (for example, <nowiki>http://ws-entry-point/codelist/all/CL_FREQ</nowiki>, could return more than one codelist if more than one agency is maintaining a codelist with id &quot;CL_FREQ&quot;).
 
* If no resourceID is specified, all matching artefacts (according to the other criteria used) should be returned. It's is therefore equivalent to using the keyword <code>all</code>.
 
* If no resourceID is specified, all matching artefacts (according to the other criteria used) should be returned. It's is therefore equivalent to using the keyword <code>all</code>.
 
* If no parameters are specified, the '''latest''' version of '''all''' resources of the type identified by the resource parameter, maintained by '''any''' maintenance agency should be returned.
 
* If no parameters are specified, the '''latest''' version of '''all''' resources of the type identified by the resource parameter, maintained by '''any''' maintenance agency should be returned.
Line 274: Line 313:
 
<li><p>To retrieve the category PRICES of the DOMAINS category scheme maintained by the ECB, as well as the categorisations referencing that category:</p>
 
<li><p>To retrieve the category PRICES of the DOMAINS category scheme maintained by the ECB, as well as the categorisations referencing that category:</p>
 
<pre>http://ws-entry-point/categoryscheme/ECB/DOMAINS/latest/PRICES?references=categorisation</pre></li>
 
<pre>http://ws-entry-point/categoryscheme/ECB/DOMAINS/latest/PRICES?references=categorisation</pre></li>
−
<li><p>To retrieve the latest version of the CL_FREQ codelists maintained by the BIS or the ECB: http://ws-entry-point/codelist/BIS+ECB/CL_FREQ</p></li></ul>
+
<li><p>To retrieve the latest version of the CL_FREQ codelists maintained by the BIS or the ECB: <nowiki>http://ws-entry-point/codelist/BIS+ECB/CL_FREQ</p></li></ul></nowiki>
  
−
== Data and Metadata Queries ==
+
= Schema queries =
  
 
=== Resources ===
 
=== Resources ===
  
−
The following resources should be supported:
+
The following resource is defined:
 +
 
 +
* schema
  
−
* data
+
This resource allows a client to ask a service to return a schema, i.e. a document which defines data validity within a certain context. The service must take into account the constraints that apply within that context (DSD or MSD, dataflow or metadataflow, or provision agreement).
−
* metadata
+
 
 +
This is typically used for validation purposes but it may also be used for communication purposes, i.e. as a way to inform providers about the data they are expected to report.
  
 
=== Parameters ===
 
=== Parameters ===
Line 289: Line 331:
 
==== Parameters used for identifying a resource ====
 
==== Parameters used for identifying a resource ====
  
−
The following parameters are used for identifying resources in data queries:
+
The following parameters are used for identifying resources:
  
 
{| class="wikitable"
 
{| class="wikitable"
 
!Parameter !! Type !! Description
 
!Parameter !! Type !! Description
 
|-
 
|-
−
| flowRef
+
| context
−
| A string identifying the dataflow. The syntax is agency id, artefact id, version, separated by a &quot;,&quot;. For example: AGENCY_ID,FLOW_ID,VERSION. In case the string only contains one out of these 3 elements, it is considered to be the flow id, i.e. all,FLOW_ID,latest. In case the string only contains two out of these 3 elements, they are considered to be the agency id and the flow id, i.e. AGENCY_ID,FLOW_ID,latest.
+
| One of the following: <code>datastructure</code>, <code>metadatastructure</code>, <code>dataflow</code>, <code>metadataflow</code>, <code>provisionagreement</code>
−
| The data (or metadata) flow of the data (or metadata) to be returned. Its a common use case in SDMX-based web services that the flow id is sufficient to uniquely identify a dataflow. Should this not be the case, the agency id and the dataflow version, can be used, in conjunction with the flow id, in order to uniquely identify a dataflow.
+
| The value of this parameter determines the constraints that need to be taken into account, when generating the schema. If datastructure or metadatastructure is used, constraints attached to the DSD or MSD must be applied when generating the schema. If dataflow or metadataflow is used, constraints attached to the dataflow or metadataflow and to the DSD or MSD used in the dataflow or metadataflow must be applied when generating the schema. If provisionagreement is used, constraints attached to the provision agreement, as well as to the dataflow or metadafalow used in the agreement and the DSD or MSD used in the dataflow or metadataflow must be applied when generating the schema.
 
|-
 
|-
−
| key
+
| agencyID
−
| A string compliant with the KeyType defined in the SDMX WADL.
+
| A string compliant with the ''SDMX common:NCNameIDType''
−
| The key of the artefact to be returned. Wildcarding is supported by omitting the dimension code for the dimension to be wildcarded. For example, if the following series key identifies the bilateral exchange rates for the daily US dollar exchange rate against the euro, D.USD.EUR.SP00.A, then the following series key can be used to retrieve the data for all currencies against the euro: D..EUR.SP00.A.The OR operator is supported using the + character. For example, the following series key can be used to retrieve the exchange rates against the euro for both the US dollar and the Japanese Yen: D.USD+JPY.EUR.SP00.A.
+
| The agency maintaining the artefact used to generate the schema to be returned.
 
|-
 
|-
−
| providerRef
+
| resourceID
−
| A string identifying the provider. The syntax is agency id, provider id, separated by a &quot;,&quot;. For example: AGENCY_ID,PROVIDER_ID. In case the string only contains one out of these 2 elements, it is considered to be the provider id, i.e. all,PROVIDER_ID.
+
| A string compliant with the SDMX ''common:IDType''
−
| The provider of the data (or metadata) to be retrieved. If not supplied, the returned message will contain data (or metadata) provided by any provider. Its a common use case in SDMX-based web services that the provider id is sufficient to uniquely identify a data provider. Should this not be the case, the agency can be used, in conjunction with the provider id, in order to uniquely identify a data provider. The OR operator is supported using the + character. For example, the following value can be used to indicate that the data should be provided by the Swiss National Bank (CH2) or Central Bank of Norway (NO2): CH2+NO2.
+
| The id of the artefact used to generate the schema to be returned.
 +
|-
 +
| version
 +
| A string compliant with the SDMX ''common:VersionType''
 +
| The version of the artefact used to generate the schema to be returned.
 
|}
 
|}
  
 
The parameters mentioned above are specified using the following syntax:
 
The parameters mentioned above are specified using the following syntax:
  
−
<pre>protocol://ws-entry-point/resource/flowRef/key/providerRef</pre>
+
<pre>protocol:// ws-entry-point/schema/context/agencyID/resourceID/version</pre>
−
Furthermore, some keywords may be used:
+
Furthermore, a keyword may be used:
  
 
{| class="wikitable"
 
{| class="wikitable"
−
!Keywords !! Scope !! Description
+
! Keyword !! Scope !! Description
 
|-
 
|-
−
| all
+
| latest
−
| key
+
| version
−
| Returns all data belonging to the specified dataflow and provided by the specified provider.
+
| Returns the latest version in production of the resource
−
|-
 
−
| all
 
−
| providerRef
 
−
| Returns all data matching the supplied key and belonging to the specified dataflow that has been provided by any data provider. As <code>all</code> is a reserved keyword in the SDMX RESTful API, it is recommended not to use it as an identifier for providers.
 
 
|}
 
|}
 +
 +
As the query for schema must match one artefact only, the keyword <code>all</code> is not supported for agencyId and resourceId.
  
 
The following rules apply:
 
The following rules apply:
  
−
* If no key is specified, all data (or metadata) belonging to the dataflow (or metadataflow) identified by the flowRef should be supplied. It is therefore equivalent to using the keyword <code>all</code>.
+
* If no version attribute is specified, the version currently used in production should be returned. It is therefore equivalent to using the keyword <code>latest</code>.
−
* If no providerRef is specified, the matching data (or metadata) provided by any data provider should be returned. It is therefore equivalent to using the keyword <code>all</code>.
 
  
−
==== Parameters used to further filter the desired results ====
+
==== Parameters used to further describe the desired results ====
  
−
The following parameters are used to further describe (or filter) the desired results, once the resource has been identified. As mentioned in 3.2, these parameters go in the query string part of the URL.
+
The following parameters are used to further describe the desired results, once the resource has been identified:
  
 
{| class="wikitable"
 
{| class="wikitable"
 
!Parameter !! Type !! Description
 
!Parameter !! Type !! Description
 
|-
 
|-
−
| startPeriod
+
| dimensionAtObservation
−
| common:StandardTimePeriodType, as defined in the SDMXCommon.xsd schema. Can be expressed using ''dateTime'' (all data that falls between the calendar dates will be matched), ''Gregorian Period'' (all data that falls between the calendar dates will be matched) or ''Reporting Period'' (all data reported as periods that fall between the specified periods will be returned. When comparing reporting weeks and days to higher order periods (e.g. quarters) one must account for the actual time frames covered by the periods to determine whether the data should be included. Data reported as Gregorian periods or distinct ranges will be returned if it falls between the specified reporting periods, based on a reporting year start day of January 1). In case the <code>:</code> or <code>+</code> characters are used, the parameter must be [http://en.wikipedia.org/wiki/URL_encoding#Percent-encoding_reserved_characters percent-encoded] by the client. Note that this value is assumed to be inclusive to the range of data being sought. For additional information about the formats, see section 4.2.14 of Section 06 (SDMX Technical Notes).
+
| A string compliant with the SDMX ''common:NCNameIDType''
−
| The start period for which results should be supplied (inclusive).
+
| The ID of the dimension to be attached at the observation level.
 
|-
 
|-
−
| endPeriod
+
| explicitMeasure
−
| Same as above
+
| ''Boolean''
−
| The end period for which results should be supplied (inclusive).
+
| For cross-sectional data validation, indicates whether observations are strongly typed (defaults to <code>false</code>).
 +
|}
 +
 
 +
=== Using an SDMX format for the response ===
 +
 
 +
By default, a ''schema'' query will return an XML schema (i.e. an <code>xsd</code> file). However, it is also possible to get the response in one of the SDMX Structure formats. In that case, the response should only include the following types of artefact: data structures, codelists, concept schemes and agency schemes. The various item schemes must only contain the following:
 +
 
 +
* For codelists, the codes that are allowed after applying the constraints up to the specified ''context'';
 +
* For concept schemes, the concepts that are used by the data structure;
 +
* For agency schemes, the agencies maintaining artefacts that are part of the response.
 +
 
 +
=== Examples ===
 +
 
 +
<ul>
 +
<li><p>To retrieve the schema for data supplied within the context of version 1.0 of the provision agreement EXR_WEB maintained by the ECB:</p>
 +
<pre>http://ws-entry-point/schema/provisionagreement/ECB/EXR_WEB/1.0/</pre>
 +
<p>In this case, the schema returned by the service must take into account the constraints attached to the provision agreement, the dataflow used in the provisionagreement and the data structure definition used in the dataflow.</p></li></ul>
 +
 
 +
=Structural Metadata Submission=
 +
Structural metadata can be added or modified by using <strong>HTTP POST</strong> to submit any valid SDMX Structure Document to the secure API.
 +
 
 +
====Resource====
 +
<pre>protocol://domain/ws/secure/sdmxapi/rest</pre>
 +
<nowiki>Example: https://registry.mydomain.org/FusionRegistry/ws/secure/sdmxapi/rest</nowiki>
 +
 
 +
====Authentication====
 +
Only authenticated users with sufficient Agency or Admin privileges can modify structural metadata content. HTTP Basic Authentication must be used to with this service.
 +
 
 +
====Actions====
 +
The following actions can be specified when submitting structural metadata content.
 +
{| class="wikitable"
 
|-
 
|-
−
| updatedAfter
+
! Action !! Behaviour
−
| xs:dateTime
 
−
| The last time the query was performed by the client in the database. If this attribute is used, the returned message should only include the latest version of what has changed in the database since that point in time (updates and revisions). This should include observations that have been added since the last time the query was performed (INSERT), observations that have been revised since the last time the query was performed (UPDATE) and observations that have been deleted since the last time the query was performed (DELETE). If no offset is specified, default to local time of the web service. If the information about when the data has been updated is not available at the observation level, the web service should return either the series that have changed (if the information is attached at the series level) or the dataflows that have changed (if the information is attached at the dataflow level).
 
 
|-
 
|-
−
| firstNObservations
+
| Append || New maintainable structures are added, but an error occurs if any of the structures submitted already exist in the Registry
−
| Positive integer
 
−
| Integer specifying the maximum number of observations to be returned for each of the matching series, starting from the first observation
 
 
|-
 
|-
−
| lastNObservations
+
| Replace || Existing maintainable structures are replaced in their entirety and new ones added if needed
−
| Positive integer
 
−
| Integer specifying the maximum number of observations to be returned for each of the matching series, counting back from the most recent observation
 
 
|-
 
|-
−
| dimensionAtObservation<sup>
+
| Merge || Similar to Append except items are added to existing item schemes - useful for adding Codes to Codelists
−
| A string compliant with the SDMX common:NCNameIDType
 
−
| The ID of the dimension to be attached at the observation level. This parameter allows the client to indicate how the data should be packaged by the service. The options are <code>TIME_PERIOD</code> (a ''timeseries'' view of the data), the <code>ID of any other dimension</code> used in that dataflow (a ''cross-sectional'' view of the data) or the keyword <code>AllDimensions</code> (a ''flat'' view of the data where the observations are not grouped, neither in time series, nor in sections). In case this parameter is not set, the service is expected to: Default to TimeDimension, if the data structure definition has one; If not, default to MeasureDimension, if the data structure definition has one; If none of the above is true, default to AllDimensions.
 
 
|-
 
|-
−
| detail
+
| FullReplace || The entire contents of the Registry are replaced with the structures submitted
−
| String
 
−
| This attribute specifies the desired amount of information to be returned. For example, it is possible to instruct the web service to return data only (i.e. no attributes). Possible options are: <code>full</code> (all data and documentation, including annotations - This is the default), <code>dataonly</code> (attributes and therefore groups will be excluded from the returned message), <code>serieskeysonly</code> (returns only the series elements and the dimensions that make up the series keys. This is useful for performance reasons, to return the series that match a certain query, without returning the actual data) and <code>nodata</code> (returns the groups and series, including attributes and annotations, without observations).
 
 
|-
 
|-
−
| includeHistory
+
| Delete || The maintainables identified in the submitted content are deleted
−
| Boolean
 
−
| This attribute allows retrieving previous versions of the data, as they were disseminated in the past (''history'' or ''timeline'' functionality). When the value is set to <code>true</code>, the returned SDMX-ML data message should contain one or two datasets per data dissemination, depending on whether a dissemination also deleted observations from the data source. The <code>validFromDate</code> and/or <code>validToDate</code> attributes of the dataset should be used to indicate the periods of validity for the data contained in the data set. See below for an example on how to handle the <code>includeHistory</code> parameter. Default to <code>false</code>.
 
 
|}
 
|}
  
−
The table below defines the meaning of parameters combinations:
+
====Submission Content Options====
−
 
+
The following SDMX Structure Documents can be submitted in the body of the POST:
 
{| class="wikitable"
 
{| class="wikitable"
−
!Combination !! Meaning
 
 
|-
 
|-
−
| startPeriod with no endPeriod
+
! Message Type !! Action
−
| Until the most recent
 
 
|-
 
|-
−
| endPeriod and no startPeriod
+
| SDMX-ML Structure Message || Specify the action using the HTTP 'Action' header (default is Replace)
−
| From the beginning
 
 
|-
 
|-
−
| startPeriod and endPeriod
+
| SDMX-JSON Structure Message || Specify the action using the HTTP 'Action' header (default is Replace)
−
| Within the supplied time range
 
 
|-
 
|-
−
| lastNObservations + startPeriod/endPeriod
+
| SDMX-EDI (GESMES) Structure Message || Specify the action using the HTTP 'Action' header (default is Replace)
−
| The specified number of observations, starting from the end, within the supplied time range
 
 
|-
 
|-
−
| firstNObservations + startPeriod/endPeriod + updatedAfter
+
| SDMX-ML SubmitStructureRequest Registry Interface Message || Specify the action using the 'Action' header in the message
−
| The specified number of observations, starting from the beginning, that have changed since the supplied timestamp, within the supplied time range
 
−
|-
 
−
| updatedAfter + startPeriod/endPeriod
 
−
| The observations, within the supplied time range, that have changed since the supplied timestamp.
 
 
|}
 
|}
  
−
=== Examples ===
+
The content can be Zipped, and multi-part submissions are also accepted.
 +
 
 +
====Response====
 +
A successful submission results in a SDMX-ML SubmitStructureResponse message.
 +
If an error occurs, a SDMX-ML ErrorResponseDocument is returned.
  
−
<ul>
+
=Structure Deletion=
−
<li><p>To retrieve the data for the series M.USD.EUR.SP00.A supplied by the ECB for the ECB_EXR1_WEB dataflow:</p>
+
The simplest way to delete Structural Metadata is using the <strong>HTTP DELETE</strong> verb.
−
<pre>http://ws-entry-point/data/ECB_EXR1_WEB/M.USD.EUR.SP00.A/ECB</pre>
 
−
<p>In this example, the assumption is made that the dataflow id (ECB_EXR1_WEB) is sufficient to uniquely identify the dataflow, and the data provider id (ECB) is sufficient to uniquely identify the data provider.</p></li>
 
−
<li><p>To retrieve the data, provided by the ECB for the ECB_EXR1_WEB dataflow, for the supplied series keys, using wildcarding for the second dimension:</p>
 
−
<pre>http://ws-entry-point/data/ECB,ECB_EXR1_WEB,latest/M..EUR.SP00.A/ECB</pre>
 
−
<p>In this example, the full reference to the dataflow is supplied (ECB as maintenance agency, ECB_EXR1_WEB as dataflow id and latest for the version)</p></li>
 
−
<li><p>To retrieve the updates and revisions for the data matching the supplied series keys, using the OR operator for the second dimension, and using percent encoding for the updatedAfter:</p>
 
−
<pre>http://ws-entry-point/Data/ECB_EXR1_WEB/M.USD+GBP+JPY.EUR.SP00.A?
 
−
updatedAfter=2009-05-15T14%3A15%3A00%2B01%3A00</pre></li>
 
−
<li><p>To retrieve the data matching the supplied series key and restricting the start and end dates:</p>
 
−
<pre>http://ws-entry-point/data/ECB_EXR1_WEB/D.USD.EUR.SP00.A?
 
−
startPeriod=2009-05-01&amp;endPeriod=2009-05-31</pre></li></ul>
 
  
−
=== Example: How to handle the <code>includeHistory</code> parameter ===
+
====Resources====
 +
Deletion of structures uses the same resources as [[REST_API_Overview#Structural_Metadata_Queries|Structural Metadata Queries]] to identify precisely what to delete.
  
−
For example, for a particular series, there were, so far, 3 disseminations: * In February 2012, there was the initial dissemination, with 2 periods: 2011-12 and 2012-01. * In March, the decision was taken to delete all observations before 2012 (so, 2011-12). In addition, a new observation has been published for 2012-02. * In April, the value for February has been revised, and the value for March has been published.
+
Maintainables
 +
<pre>protocol://domain/ws/secure/sdmxapi/rest/resource/agencyID/resourceID/version</pre>
 +
Example:<br>
 +
<nowiki>https://registry.mydomain.org/ws/secure/sdmxapi/rest/datastructure/ECB/ECB_TRD/1.0</nowiki><br>
  
−
If the value of the includeHistory is set to true, the web service should return 4 datasets: * The first dataset contains the data disseminated in February, so 2 observations (2011-12 and 2012-01). The dataset action flag is <code>Replace</code>. * The second dataset contains the new data disseminated in March. It will contain one observation (2012-02). The dataset action flag is also <code>Replace</code>. * The third dataset contains the deleted data, removed with the March dissemination. It will contain one observation (2011-12). The dataset action flag is <code>Delete</code>. * The fourth dataset contains the data disseminated in April. It will contain the revised observation (2012-02) and the new one (2012-03). The dataset action flag is <code>Replace</code>.
+
Items
 +
<pre>protocol://domain/ws/secure/sdmxapi/rest/resource/agencyID/resourceID/version/itemID</pre>
 +
Examples:<br>
 +
<nowiki>https://registry.mydomain.org/ws/secure/sdmxapi/rest/codelist/ECB/CL_FREQ/1.0/A</nowiki><br>
 +
<nowiki>https://registry.mydomain.org/ws/secure/sdmxapi/rest/codelist/ECB/CL_FREQ/1.0/A+M+Q</nowiki><br>
  
−
The <code>validFrom</code> and <code>validTo</code> flags should be used as follows: * For datasets whose action flag is <code>Replace</code>, the <code>validFromDate</code> is used to indicate from which point in time the values are considered valid. * For datasets whose action flag is <code>Delete</code>, the <code>validToDate</code> is used to indicate until which point in time the values were considered valid.
+
====Authentication====
 +
Only authenticated users with sufficient Agency or Admin privileges can modify structural metadata content. HTTP Basic Authentication must be used to with this service.
  
−
=Fusion Metadata Registry Extended API=
+
====Response====
 +
A successful submission results in a SDMX-ML SubmitStructureResponse message.
 +
If an error occurs, a SDMX-ML ErrorResponseDocument is returned.

Latest revision as of 23:55, 4 July 2022

SDMX Standard REST API Specification

Scope of the SDMX REST API

The RESTful API focuses on simplicity. The aim is not to replicate the full semantic richness of the SDMX-ML Query message but to make it simple to perform a limited set of standard queries. Also, in contrast to other parts of the SDMX specification, the RESTful API focuses solely on data retrieval (via HTTP GET). More specifically, the API allows:

  • To retrieve structural metadata, using a combination of id, agencyID and version number.
  • To retrieve statistical data or reference metadata using keys (with options for wildcarding and support for the OR operator), data or metadata flows and data or metadata providers.
  • To further refine queries for statistical data or reference metadata using time information (start period and end period).
  • To retrieve updates and revisions only.
  • To return the results of a query in various formats. The desired format and version of the returned message will be specified using HTTP Content Negotiation (and the HTTP Accept request header).
  • For structural metadata, it is possible to instruct the web service to resolve references (for instance, when querying for data structure definitions, it is possible to also retrieve the concepts and code lists used in the returned data structure definitions), as well as artefacts that use the matching artefact (for example, to retrieve the dataflows that use a matching data structure definition).
  • For structural metadata, it is possible to retrieve a minimal version of the artefact, for the sake of efficiency (for example, to retrieve all code lists – names, ids, etc – without the codes).
  • A distinction should be established between the elements that allow identifying the resource to be retrieved and the elements that give additional information about, or allow to further filter, the desired results. Elements belonging to the 1st category are specified in the path part of the URL while elements belonging to the 2nd category are specified in the query string part of the URL.

Implementation of the SDMX REST API in Fusion Metadata Registry

Fusion Metadata Registry is primarily concerned with structural metadata so implements a subset of the SDMX RESTful API specification.

SDMX REST API Function Implemented
Structural Metadata Queries YES
Schema Queries (for retrieving SDMX-ML Structure Specific Data validation schemas) YES
Data Queries NO
Data Availability NO

The Fusion Metadata Registry API also provides some additional functions that are necessary for a practical registry implementation, but not explicitly provided for in SDMX standard REST specification.

FMR extended REST API Function Purpose
Structure Submission The addition or modification of structures using HTTP POST
Structure Deletion Delete specific structures using HTTP DELETE
Data Validation (synchronous) Validates a dataset submitted using HTTP POST returning a validation report in JSON
Data Transformation (synchronous) Transforms a dataset submitted using HTTP POST to a different DSD with an optional report returned in JSON
Data Transformation and Validation (asynchronous) Validates and transforms datasets asynchronously making it suitable for larger submissions
Product Info Returns details about the running Fusion Metadata Registry Instance in JSON
Active User Returns details about the current authenticated user in JSON

Structural Metadata Queries

A structural metadata query uses HTTP GET with the following general URL syntax: 

http(s)://ws-entry-point/resource/agencyID/resourceID/version

Resources

The following resources are defined:

  • datastructure (This has been shortened from DataStructureDefinition to allow for shorter URLs)
  • metadatastructure (This has been shortened from MetadataStructureDefinition to allow for shorter URLs)
  • categoryscheme
  • conceptscheme
  • codelist
  • hierarchicalcodelist
  • organisationscheme (The organisationscheme resource can be used whenever the role played by the organisation schemes is not known/relevant)
  • agencyscheme (For 3 of the subtypes of OrganisationScheme, the id and version parameters have fixed values. See Section 03 of the SDMX information model document for additional information)
  • dataproviderscheme
  • dataconsumerscheme
  • organisationunitscheme
  • dataflow
  • metadataflow
  • reportingtaxonomy
  • provisionagreement
  • structureset
  • process
  • categorisation
  • contentconstraint
  • actualconstraint (a type of contentconstraint stating what data are actually present)
  • allowedconstraint (a type of contentconstraint defining what data are allowed)
  • attachmentconstraint
  • transformationscheme
  • rulesetscheme
  • userdefinedoperatorscheme
  • customtypescheme
  • namepersonalisationscheme
  • vtlmappingscheme
  • structure (This type can be used to retrieve any type of structural metadata matching the supplied parameters)

Parameters

Parameters used for identifying a resource

The following parameters are used for identifying resources:

Parameter Type Description
agencyID A string compliant with the SDMX common:NCNameIDType The agency maintaining the artefact to be returned. It is possible to set more than one agency, using + as separator (e.g. BIS+ECB).
resourceID A string compliant with the SDMX common:IDType The id of the artefact to be returned. It is possible to set more than one id, using + as separator (e.g. CL_FREQ+CL_CONF_STATUS).
version A string compliant with the SDMX common:VersionType The version of the artefact to be returned. It is possible to set more than one version, using + as separator (e.g. 1.0+2.1).

The parameters mentioned above are specified using the following syntax: 

protocol://ws-entry-point/resource/agencyID/resourceID/version

Furthermore, some keywords may be used:

Keywords Scope Description
all agencyID Returns artefacts maintained by any maintenance agency
all resourceID Returns all resources of the type defined by the resource parameter
all version Returns all versions of the resource
latest version Returns the latest version in production of the resource

As all is a reserved keyword in the SDMX RESTful API, it is recommended not to use it as an identifier for agencies, resources or a specific version.

The following rules apply:

  • If no version is specified, the version currently used in production should be returned. It is therefore equivalent to using the keyword latest.
  • If no agencyID is specified, the matching artefacts maintained by any maintenance agency should be returned. It is therefore equivalent to using the keyword all. This would potentially return more than one artefact, if different agencies give the same identifier to a resource (for example, http://ws-entry-point/codelist/all/CL_FREQ, could return more than one codelist if more than one agency is maintaining a codelist with id "CL_FREQ").
  • If no resourceID is specified, all matching artefacts (according to the other criteria used) should be returned. It's is therefore equivalent to using the keyword all.
  • If no parameters are specified, the latest version of all resources of the type identified by the resource parameter, maintained by any maintenance agency should be returned.

Additional parameter used for identifying a resource, for item scheme types

SDMX uses the item scheme pattern to model SDMX collections of items. These are:

  • categoryscheme
  • conceptscheme
  • codelist
  • organisationscheme
  • agencyscheme
  • dataproviderscheme
  • dataconsumerscheme
  • organisationunitscheme
  • reportingtaxonomy
  • transformationscheme
  • rulesetscheme
  • userdefinedoperatorscheme
  • customtypescheme
  • namepersonalisationscheme
  • vtlmappingscheme

Although it is not following the item scheme pattern, hierarchicalcodelist is also a collection, i.e. a collection of hierarchies.

For these collections (those following the item scheme pattern or the hierarchicalcodelist), it is possible to use a 4th parameter for identifying a resource. The rules for the 3 other parameters, as defined in the section above, remain valid.

Parameter Type Description
itemID A string compliant with the SDMX common:NestedNCNameIDType for conceptscheme and agencyscheme, SDMX common:IDType for hierarchicalcodelist or with the SDMX common:NestedIDType in all other cases The id of the item to be returned. It is possible to set more than one id, using + as separator (e.g. A+Q+M).

This 4th parameter is used as follows: 

protocol://ws-entry-point/resource/agencyID/resourceID/version/itemID

Furthermore, a keyword may be used:

Keyword Scope Description
all itemID Returns all items belonging to the item scheme

The following rules apply:

  • If no itemID is specified, all the items belonging to the item scheme should be returned. It is therefore equivalent to using the keyword all.
  • If itemID is set and is a top-level id (e.g.: Code A (Annual) in the Frequency Codelist), and such an item exists in the matching item scheme, the item scheme returned should contain only the matching item and its isPartial parameter should be set to true.
  • If itemID is set and is a nested id (e.g.: Category A.1.1, belonging to Category A.1, belonging to Category A in a Category Scheme), and such an item exists in the matching item scheme, the item scheme returned should contain the matching item and its ancestors, and its isPartial parameter should be set to true.

Parameters used to further describe the desired results

The following parameters are used to further describe the desired results, once the resource has been identified. As mentioned in 3.2, these parameters appear in the query string part of the URL.

Parameter Type Description Default
detail String This attribute specifies the desired amount of information to be returned. For example, it is possible to instruct the web service to return only basic information about the maintainable artefact (i.e.: id, agency id, version and name). Most notably, items of item schemes will not be returned (for example, it will not return the codes in a code list query). Possible values are: allstubs (all artefacts should be returned as stubs, containing only identification information, as well as the artefacts' name), referencestubs (referenced artefacts should be returned as stubs, containing only identification information, as well as the artefacts' name), referencepartial (referenced item schemes should only include items used by the artefact to be returned. For example, a concept scheme would only contain the concepts used in a DSD, and its isPartial flag would be set to true. Likewise, if a dataflow has been constrained, then the codelists referenced by the DSD referenced by the dataflow should only contain the codes allowed by the content constraint), allcompletestubs (all artefacts should be returned as complete stubs, containing identification information, the artefacts' name, description, annotations and isFinal information), referencecompletestubs (referenced artefacts should be returned as complete stubs, containing identification information, the artefacts' name, description, annotations and isFinal information) and full (all available information for all artefacts should be returned). full
references String This attribute instructs the web service to return (or not) the artefacts referenced by the artefact to be returned (for example, the code lists and concepts used by the data structure definition matching the query), as well as the artefacts that use the matching artefact (for example, the dataflows that use the data structure definition matching the query). Possible values are: none (no references will be returned), parents (the artefacts that use the artefact matching the query), parentsandsiblings (the artefacts that use the artefact matching the query, as well as the artefacts referenced by these artefacts), children (artefacts referenced by the artefact to be returned), descendants (references of references, up to any level, will also be returned), all (the combination of parentsandsiblings and descendants). In addition, a concrete type of resource may also be used (for example, references=codelist). none

Applicability and meaning of references attribute

The table below lists the 1st level artefacts (one level up, one level down) that will be returned if the references parameter is set to all. Artefacts referenced by the matching artefact are displayed in regular style, while the artefacts that reference the matching artefact are displayed in Italic.

Maintainable artefact Artefacts returned
AgencyScheme Categorisation, Process, MetadataStructureDefinition, StructureSet
Categorisation All
CategoryScheme Categorisations, Process, StructureSet
Codelist Categorisation, Process, HierarchicalCodelist, ConceptScheme, DataStructureDefinition, MetadataStructureDefinition, StructureSet, VtlMappingScheme
ConceptScheme Categorisation, Process, Codelist, DataStructureDefinition, MetadataStructureDefinition, StructureSet, VtlMappingScheme
Constraint Categorisation, Process, DataProviderScheme, DataStructureDefinition, Dataflow, MetadataStructureDefinition, Metadataflow, ProvisionAgreement
DataConsumerScheme Categorisation, Process, MetadataStructureDefinition, StructureSet
Dataflow Categorisation, Process, Constraint, DataStructureDefinition, ProvisionAgreement, ReportingTaxonomy, StructureSet, VtlMappingScheme
DataProviderScheme Categorisation, Process, Constraint, ProvisionAgreement, MetadataStructureDefinition, StructureSet
DataStructureDefinition Categorisation, Process, Codelist, ConceptScheme, Constraint, Dataflow, StructureSet
HierarchicalCodelist Categorisation, Process, Codelist, StructureSet
Metadataflow Categorisation, Process, Constraint, MetadataStructureDefinition, ProvisionAgreement, ReportingTaxonomy, StructureSet
MetadataStructureDefinition Categorisation, Process, ConceptScheme, Codelist, DataProviderScheme, DataConsumerScheme, AgencyScheme, OrganisationUnitScheme, Constraint, Metadataflow, StructureSet
OrganisationUnitScheme Categorisation, Process, Constraint, MetadataStructureDefinition, StructureSet
Process All
ProvisionAgreement Categorisation, Process, DataProviderScheme, Dataflow, Metadataflow, Constraint
ReportingTaxonomy Categorisation, Process, Dataflow, Metadataflow, StructureSet
StructureSet Categorisation, Process, DataStructureDefinition, MetadataStructureDefinition, CategoryScheme, DataProviderScheme, DataConsumerScheme, AgencyScheme, OrganisationUnitScheme, ConceptScheme, Codelist, ReportingTaxonomy, HierarchicalCodelist, Dataflow, Metadataflow
CustomTypeScheme AgencyScheme, Categorisation, TranformationScheme
NamePersonalisationScheme AgencyScheme, Categorisation, TranformationScheme
RulesetScheme AgencyScheme, Categorisation, TranformationScheme, VtlMappingScheme
TranformationScheme AgencyScheme, Categorisation, CustomTypeScheme, NamePersonalisationScheme, RulesetScheme, UserDefinedOperatorScheme, VtlMappingScheme
UserDefinedOperatorScheme AgencyScheme, Categorisation, TranformationScheme, VtlMappingScheme
VtlMappingScheme AgencyScheme, Categorisation, Codelist, ConceptScheme, Dataflow, RulesetScheme, TranformationScheme, UserDefinedOperatorScheme

Examples

  • To retrieve version 1.0 of the DSD with id ECB_EXR1 maintained by the ECB, as well as the code lists and the concepts used in the DSD:

    http://ws-entry-point/datastructure/ECB/ECB_EXR1/1.0?references=children&detail=referencepartial

  • To retrieve the latest version in production of the DSD with id ECB_EXR1 maintained by the ECB, without the code lists and concepts of the DSD:

    http://ws-entry-point/datastructure/ECB/ECB_EXR1

  • To retrieve all DSDs maintained by the ECB, as well as the dataflows using these DSDs:

    http://ws-entry-point/datastructure/ECB?references=dataflow

  • To retrieve the latest version in production of all code lists maintained by all maintenance agencies, but without the codes:

    http://ws-entry-point/codelist?detail=allstubs

  • To retrieve, as stubs, the latest version in production of all maintainable artefacts maintained by the ECB:

    http://ws-entry-point/structure/ECB?detail=allstubs

  • To retrieve the category PRICES of the DOMAINS category scheme maintained by the ECB, as well as the categorisations referencing that category:

    http://ws-entry-point/categoryscheme/ECB/DOMAINS/latest/PRICES?references=categorisation

  • To retrieve the latest version of the CL_FREQ codelists maintained by the BIS or the ECB: http://ws-entry-point/codelist/BIS+ECB/CL_FREQ</p></li></ul>

    Schema queries

    Resources

    The following resource is defined:

    • schema

    This resource allows a client to ask a service to return a schema, i.e. a document which defines data validity within a certain context. The service must take into account the constraints that apply within that context (DSD or MSD, dataflow or metadataflow, or provision agreement).

    This is typically used for validation purposes but it may also be used for communication purposes, i.e. as a way to inform providers about the data they are expected to report.

    Parameters

    Parameters used for identifying a resource

    The following parameters are used for identifying resources:

    Parameter Type Description
    context One of the following: datastructure, metadatastructure, dataflow, metadataflow, provisionagreement The value of this parameter determines the constraints that need to be taken into account, when generating the schema. If datastructure or metadatastructure is used, constraints attached to the DSD or MSD must be applied when generating the schema. If dataflow or metadataflow is used, constraints attached to the dataflow or metadataflow and to the DSD or MSD used in the dataflow or metadataflow must be applied when generating the schema. If provisionagreement is used, constraints attached to the provision agreement, as well as to the dataflow or metadafalow used in the agreement and the DSD or MSD used in the dataflow or metadataflow must be applied when generating the schema.
    agencyID A string compliant with the SDMX common:NCNameIDType The agency maintaining the artefact used to generate the schema to be returned.
    resourceID A string compliant with the SDMX common:IDType The id of the artefact used to generate the schema to be returned.
    version A string compliant with the SDMX common:VersionType The version of the artefact used to generate the schema to be returned.

    The parameters mentioned above are specified using the following syntax: 

    protocol:// ws-entry-point/schema/context/agencyID/resourceID/version

    Furthermore, a keyword may be used:

    Keyword Scope Description
    latest version Returns the latest version in production of the resource

    As the query for schema must match one artefact only, the keyword all is not supported for agencyId and resourceId.

    The following rules apply:

    • If no version attribute is specified, the version currently used in production should be returned. It is therefore equivalent to using the keyword latest.

    Parameters used to further describe the desired results

    The following parameters are used to further describe the desired results, once the resource has been identified:

    Parameter Type Description
    dimensionAtObservation A string compliant with the SDMX common:NCNameIDType The ID of the dimension to be attached at the observation level.
    explicitMeasure Boolean For cross-sectional data validation, indicates whether observations are strongly typed (defaults to false).

    Using an SDMX format for the response

    By default, a schema query will return an XML schema (i.e. an xsd file). However, it is also possible to get the response in one of the SDMX Structure formats. In that case, the response should only include the following types of artefact: data structures, codelists, concept schemes and agency schemes. The various item schemes must only contain the following:

    • For codelists, the codes that are allowed after applying the constraints up to the specified context;
    • For concept schemes, the concepts that are used by the data structure;
    • For agency schemes, the agencies maintaining artefacts that are part of the response.

    Examples

    • To retrieve the schema for data supplied within the context of version 1.0 of the provision agreement EXR_WEB maintained by the ECB:

      http://ws-entry-point/schema/provisionagreement/ECB/EXR_WEB/1.0/

      In this case, the schema returned by the service must take into account the constraints attached to the provision agreement, the dataflow used in the provisionagreement and the data structure definition used in the dataflow.

    Structural Metadata Submission

    Structural metadata can be added or modified by using HTTP POST to submit any valid SDMX Structure Document to the secure API.

    Resource

    protocol://domain/ws/secure/sdmxapi/rest

    Example: https://registry.mydomain.org/FusionRegistry/ws/secure/sdmxapi/rest

    Authentication

    Only authenticated users with sufficient Agency or Admin privileges can modify structural metadata content. HTTP Basic Authentication must be used to with this service.

    Actions

    The following actions can be specified when submitting structural metadata content.

    Action Behaviour
    Append New maintainable structures are added, but an error occurs if any of the structures submitted already exist in the Registry
    Replace Existing maintainable structures are replaced in their entirety and new ones added if needed
    Merge Similar to Append except items are added to existing item schemes - useful for adding Codes to Codelists
    FullReplace The entire contents of the Registry are replaced with the structures submitted
    Delete The maintainables identified in the submitted content are deleted

    Submission Content Options

    The following SDMX Structure Documents can be submitted in the body of the POST:

    Message Type Action
    SDMX-ML Structure Message Specify the action using the HTTP 'Action' header (default is Replace)
    SDMX-JSON Structure Message Specify the action using the HTTP 'Action' header (default is Replace)
    SDMX-EDI (GESMES) Structure Message Specify the action using the HTTP 'Action' header (default is Replace)
    SDMX-ML SubmitStructureRequest Registry Interface Message Specify the action using the 'Action' header in the message

    The content can be Zipped, and multi-part submissions are also accepted.

    Response

    A successful submission results in a SDMX-ML SubmitStructureResponse message. If an error occurs, a SDMX-ML ErrorResponseDocument is returned.

    Structure Deletion

    The simplest way to delete Structural Metadata is using the HTTP DELETE verb.

    Resources

    Deletion of structures uses the same resources as Structural Metadata Queries to identify precisely what to delete.

    Maintainables 

    protocol://domain/ws/secure/sdmxapi/rest/resource/agencyID/resourceID/version

    Example:
    https://registry.mydomain.org/ws/secure/sdmxapi/rest/datastructure/ECB/ECB_TRD/1.0

    Items 

    protocol://domain/ws/secure/sdmxapi/rest/resource/agencyID/resourceID/version/itemID

    Examples:
    https://registry.mydomain.org/ws/secure/sdmxapi/rest/codelist/ECB/CL_FREQ/1.0/A
    https://registry.mydomain.org/ws/secure/sdmxapi/rest/codelist/ECB/CL_FREQ/1.0/A+M+Q

    Authentication

    Only authenticated users with sufficient Agency or Admin privileges can modify structural metadata content. HTTP Basic Authentication must be used to with this service.

    Response

    A successful submission results in a SDMX-ML SubmitStructureResponse message. If an error occurs, a SDMX-ML ErrorResponseDocument is returned.

Retrieved from "https://fmrwiki.sdmxcloud.org/index.php?title=REST_API_Overview&oldid=5710"