Data Validation Web Service - Revision history

Mnelson: /* HTTP Headers */

2022-12-21T12:49:03Z

‎HTTP Headers

Vmurrell: /* Dataset which could not be read */

2021-03-27T10:25:19Z

‎Dataset which could not be read

Vmurrell: /* Dataset with Errors */

2021-03-27T10:25:00Z

‎Dataset with Errors

Vmurrell: /* HTTP Headers */

2021-03-27T10:24:13Z

‎HTTP Headers

Glenn at 10:04, 24 September 2020

2020-09-24T10:04:33Z

Mnelson: /* HTTP Headers */

2020-08-18T20:43:52Z

‎HTTP Headers

Mnelson: /* HTTP Headers */

2020-08-18T20:43:39Z

‎HTTP Headers

Mnelson: /* HTTP Headers */

2020-03-19T09:49:41Z

‎HTTP Headers

Mnelson: Created page with "Category:WebService = Overview = {| class="wikitable" |- |style="background-color:#eaecf0"|Entry Point|| /ws/public/data/validate |- |style="background-col..."

2020-03-02T14:32:02Z

Created page with "Category:WebService = Overview = {| class="wikitable" |- |style="background-color:#eaecf0"|Entry Point|| /ws/public/data/validate |- |style="background-col..."

New page

[[Category:WebService]]
= Overview =
{| class="wikitable"
|-
|style="background-color:#eaecf0"|Entry Point|| /ws/public/data/validate 
|-
|style="background-color:#eaecf0"|Access|| Public (default). Configurable to Private
|-
|style="background-color:#eaecf0"|Http Method|| POST
|-
|style="background-color:#eaecf0"|Accepts|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)
|-
|style="background-color:#eaecf0"|Compression || Zip files supported, if loading from URL gzip responses supported
|-
|style="background-color:#eaecf0"|Content-Type|| 1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile
2. application/text or application/xml (if submitting data in the body of the POST)
|-
|style="background-color:#eaecf0"|Response Format|| application/json
|-
|style="background-color:#eaecf0"|Response Statuses|| 200 - Validation could be performed
400 - Validation could not be performed (either an unreadable dataset, or unresolvable reference to a required structure)
401 - Unauthorized (if access has been restricted)
500 - Server Error
|}

= HTTP Headers =
{| class="wikitable"
|-
!|HTTP Header || Purpose || Allowed Values
|-
|style="background-color:#eaecf0"|Data-Format|| Used to inform the server when the data is in CSV format. ||csv;delimiter=[delimiter]
Where [delimiter] is either:
<ul>
<li>comma</li>
<li>tab</li>
<li>semicolon</li>
<li>space</li>
</ul>
|-
|style="background-color:#eaecf0"|Structure || (optional) Provides the structure to validate the data against.

This is optional as this information may be present in the header of the DataSet. If provided this value will override the value in the dataset (if present).
|| Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition
|-
|style="background-color:#eaecf0"|Inc-Metrics(Since v9.8)||
Optional. Includes metrics on the validation.
This will add extra detail to the validation report.
||Boolean (true/false)
|-
|style="background-color:#eaecf0"|Inc-Valid(Since v9.8)||Optional. Instructs the service to include a dataset with all the valid series and observations in the response.

As the result will contain a separate file for the dataset, the response format will be set to either multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.
The file is called ValidData with the file extension based on the output format.
||Boolean (true/false)

|-
|style="background-color:#eaecf0"|Inc-Invalid(Since v9.8)||
Optional. Instructs the service to include a dataset with all the invalid series and observations in the response. 

As the result will contain a separate file for the dataset, the response format will be set to either multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.
The file is called InvalidData with the file extension based on the output format.

||Boolean (true/false)
|-
|style="background-color:#eaecf0"|Accept(Since v9.8)||Optional. Instructs the service which data output format to output the valid or invalid datasets in.

This Header is only used if Inc-Valid or Inc-Invalid are set to true.
||See Accept formats for REST Data Query
|-
|style="background-color:#eaecf0"|Zip(Since v9.8)||Optional. Compresses the output as a zip file. If used in conjunction with Inc-Valid or Inc-Invalid the zip will contain multiple files.
||Boolean (true/false)
|-
|style="background-color:#eaecf0"|Prior-Data-Dependent(Since v9.8)||
Optional. This allows data to be validated under the assumption that other data will provide missing information. If this value is set to true, particular data validators will not be used when validating the data. These validators are "Mandatory Observations" and "Valid Calculations".

Default value is false.

||Boolean (true/false)
|}

= Response =

The validation output contains both human readable error descriptions, as well as machine processible locations of the errors within the dataset. The location in the dataset is described as a key or observation locator in the format; A:UK:M:2008 – where each component relates to the Dimension value, separated by a colon. If the error position is observation, the last part of the key is the observation time period.

There are 3 types of output that can be produced which share a common structure: unable to parse input(returns HTTP 400); able to parse input but references invalid data structure (returns HTTP 200); parsed input and returns output, which may have validation errors (return HTTPS 200). Below are examples of each:

== Valid Dataset ==

{
"Meta": {
"RequestTime": 1564410081711,
"Duration": 43
},
"FileFormat": "Structure Specific (Compact) v2.1",
"Prepared": "2019-07-29T10:23:01",
"SenderId": "FR_DEMO",
"DataSetId": null,
"Status": "Complete",
"Errors": false,
"Datasets": [
{
"DSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=OECD:HIGH_AGLINK_2011(1.0)",
"Dataflow": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:AGRIC_OUTLOOK_2011_2020(1.0)",
"DataProvider": "urn:sdmx:org.sdmx.infomodel.base.DataProvider=METATECH:DATA_PROVIDERS(1.0).METATECH",
"ProvisionAgreement": "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)",
"KeysCount": 2,
"ObsCount": 62,
"GroupsCount": 0,
"Errors": false
"ReportedPeriods": {
"A": {
"Name": "Annual",
"StartPeriod": "1990",
"EndPeriod": "2020"
}
},
}
],
"PreventsConversion": false,
"PreventsPublication": false
}

== Dataset with Errors ==
{
"Meta": {
"RequestTime": 1564401209760,
"Duration": 34
},
"InvalidData": {
"Datasets": [
{
"Structure": "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)",
"Series": 2,
"Observations": 61,
"Groups": 0
}
]
},
"ValidData": {
"Datasets": [
{
"Structure": "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)",
"Series": 2,
"Observations": 32,
"Groups": 0
}
]
},
"FileFormat": "Structure Specific (Compact) v2.1",
"Prepared": "2019-07-29T10:23:01",
"SenderId": "FR_DEMO",
"DataSetId": null,
"Status": "Complete",
"Errors": true,
"Datasets": [
{
"DSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=OECD:HIGH_AGLINK_2011(1.0)",
"Dataflow": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:AGRIC_OUTLOOK_2011_2020(1.0)",
"DataProvider": "urn:sdmx:org.sdmx.infomodel.base.DataProvider=METATECH:DATA_PROVIDERS(1.0).METATECH",
"ProvisionAgreement": "urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)",
"KeysCount": 3,
"ObsCount": 93,
"GroupsCount": 0,
"ReportedPeriods": {
"A": {
"Name": "Annual",
"StartPeriod": "1990",
"EndPeriod": "2020"
}
},
"Errors": true,
"ValidationReport": [
{
"Type": "Constraint",
"Errors": [
{
"Message": "Disallowed Dimension Value: REF_AREA=AFR",
"Dataset": 0,
"ComponentId": " REF_AREA ",
"ReportedValue": "AFR",
"Position": "Series",
"Keys": ["AFR:BT:AA"]
}
]
},
{
"Type": "Representation",
"Errors": [
{
"Message": "Dimension 'VARIABLE' is reporting value 'AA' which is not a valid representation in referenced Codelist 'OECD:CL_HIGH_AGLINK_2011_VARIABLE(1.0)'",
"Dataset": 0,
"Position": "Series",
"ComponentId": "VARIABLE",
"ReportedValue": "AA",
"Keys": ["AFR:BT:AA"]
},
{
"Message": "Error in Primary Measure 'OBS_VALUE': Reported value 'XXX' is not of expected type 'Double'",
"Dataset": 0,
"ComponentId": " OBS_VALUE",
"ReportedValue": "XXX",
"Position": "Observation",
"Keys": ["AFR:BT:IM:2010"]
}
]
},
{
"Type": "FormatSpecific",
"Errors": [
{
"Message": "Unexpected attribute 'ASD' for element 'StructureSpecificData/DataSet/Series/Obs'",
"Dataset": 0,
"Position": "Dataset"
}
]
}
]
}
],
"PreventsConversion": false,
"PreventsPublication": true
}

Note the first three elements ‘Meta’, ‘InvalidData’, ‘ValidData’, there are present in the report if Inc-Metrics is set to true. Inc-valid and Inc-Invalid set to true enables the report to know the metrics for the invalid and valid data.
Note also each Error has a Type, this is the category of error which caused the validator to fail. For a list of all validators see the following section on Validators.
The Error Position is either set to Dataset, Series, Observation, or Group. 
PreventsConversion and PreventsPublication is an indication on the severity of the error. These settings on which errors prevent conversion and publication can be set in the Fusion Registry by the administrator of the system.

== Dataset with an Unresolvable Datset Reference ==
{
"FileFormat": "Generic v2.1",
"MimeType": "application/xml",
"Status": "InvalidRef",
"Errors": true,
"Datasets": [{"Dataflow": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:INVALID_DATAFLOW(1.0)"}]
}
Errors has a value of true, the Status states InvalidRef, and the Datasets provides the reported reference which could not be resolved

== Dataset which could not be read ==
{
"Status": "Error",
"Errors": true,
"Error": "Unexpected '<' character in element (missing closing '>'?)\r\n at [row,col {unknown-source}]: [17,3]"
}
This error will be reported when the Fusion Registry is unable to determine what type of data the dataset is, so is unable to process the dataset for validation

@@ Line 67: / Line 67: @@
 |style="background-color:#eaecf0"|<b>Accept</b><p style="font-size:0.85em">(Since v9.8)</p>||<p>Optional. Instructs the service which data output format to output the valid or invalid datasets in.
 </p>
-<p>This Header is only used if Inc-Valid or Inc-Invalid are set to true.</p>
+<p>'''Note''': From FMR 11.5.0 the format (if not specified) defaults to the input format.  Previous versions defaulted to SDMX Structure Specific 2.1</p>
 ||[[Data_Formats|See Accept Formats]]
 |-

@@ Line 246: / Line 246: @@
     "Error": "Unexpected '<' character in element (missing closing '>'?)\r\n at [row,col {unknown-source}]: [17,3]"
+  }
-<p>This error will be reported when the Fusion Registry is unable to determine what type of data the dataset is, so is unable to process the dataset for validation</p>
+<p>This error will be reported when the Fusion Metadata Registry is unable to determine what type of data the dataset is, so is unable to process the dataset for validation</p>

@@ Line 228: / Line 228: @@
 <p>Note also each Error has a Type, this is the category of error which caused the validator to fail.  For a list of all validators see the following section on Validators.
 The Error Position is either set to Dataset, Series, Observation, or Group.  </p>
-<p>PreventsConversion and PreventsPublication is an indication on the severity of the error.  These settings on which errors prevent conversion and publication can be set in the Fusion Registry by the administrator of the system.</p>
+<p>PreventsConversion and PreventsPublication is an indication on the severity of the error.  These settings on which errors prevent conversion and publication can be set in the Fusion Metadata Registry by the administrator of the system.</p>
 == Dataset with an Unresolvable Datset Reference ==

@@ Line 41: / Line 41: @@
 <p>This information may be present in the header of the dataset, if provided this value will override the value in the dataset.</p>
-<p>Fusion Registry v10.4.7 onwards accepts a comma separated list of URNs if multiple datasets exist in the loaded file, however if multiple URNs are provided, URN must be of a Dataflow or Provision Agreement and the datasets must identify at a minimum the Data Structure that is needed to read the file.</p>
+<p>Fusion Metadada Registry v10.4.7 onwards accepts a comma separated list of URNs if multiple datasets exist in the loaded file, however if multiple URNs are provided, URN must be of a Dataflow or Provision Agreement and the datasets must identify at a minimum the Data Structure that is needed to read the file.</p>
 || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition

@@ Line 41: / Line 41: @@
 <p>This information may be present in the header of the dataset, if provided this value will override the value in the dataset.</p>
-<p>Fusion Registry v10.4.7 accepts a comma separated list of URNs if multiple datasets exist in the loaded file, however if multiple URNs are provided, URN must be of a Dataflow or Provision Agreement and the datasets must identify at a minimum the Data Structure that is needed to read the file.</p>
+<p>Fusion Registry v10.4.7 onwards accepts a comma separated list of URNs if multiple datasets exist in the loaded file, however if multiple URNs are provided, URN must be of a Dataflow or Provision Agreement and the datasets must identify at a minimum the Data Structure that is needed to read the file.</p>
 || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition

@@ Line 1: / Line 1: @@
-[[Category:WebService]]
+[[Category:FMR REST API Reference]]
 = Overview =
 {| class="wikitable"

@@ Line 40: / Line 40: @@
 |style="background-color:#eaecf0"|<b>Structure</b> || <p>(optional) Provides the structure to validate the data against.</p>
-<p>This is optional as this information may be present in the header of the DataSet.  If provided this value will override the value in the dataset (if present).</p>
+<p>This information may be present in the header of the dataset, if provided this value will override the value in the dataset.</p>
 || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition
 |-

@@ Line 66: / Line 66: @@
 </p>
 <p>This Header is only used if Inc-Valid or Inc-Invalid are set to true.</p>
-||See Accept formats for REST Data Query
+||[[Data_Formats|See Accept Formats]]
 |-
 |style="background-color:#eaecf0"|<b>Zip</b><p style="font-size:0.85em">(Since v9.8)</p>||Optional. Compresses the output as a zip file.  If used in conjunction with <b>Inc-Valid</b> or <b>Inc-Invalid</b> the zip will contain multiple files.