Explain Plan Web Services

From FMR Knowledge Base
Jump to navigation Jump to search

Overview

In the UI of the FMR, on the page "Structure Maps" is the control "Test Mapping" which can be used to test an input file against a Mapping. This feature is very useful when debugging why a mapping did not occur. Mapping is explained in deatil on the Structure Map page. This page details this functionality and how to directly call the Web Services that perform the explanation of a mapping.

Using the User Interface

Navigate to the Structure Map page and select the Structure Map to test. Click on the orange "Test Mapping" button. A modal window appears requesting which DSD is to be used as the source and what data file to test. A data file must be loaded which conforms to either the source or the target DSD. Then click "Start Test".

In the subsequent modal that is displayed, the mapping report is shown. Mapping reports are generated at the level of each row of information. If loading time series data, a row should be thought of as a single observation along with all the series information. The report will describe the output row for each input row.

FMR can perform finer-grained reports detailing exactly how a specific row was mapped. Using the User Interface this is achieved by selecting a row. The mapping report is broken-down into each mapping rule within the underlying Structur Map, what the input was for the rule, and what the output was.

The Web Services

There are 2 Web Services. The first "explains" what series dimensions of an input file would be mapped and which would be unmapped with regards to a specifc Map. The second Web Service analyses a particular Series Key.

'Explain Map' Web Service

From the supplied file and the specified mapping structure URN, returns a JSON object stating which series were mapped and which were unmapped from the input.

Entry Point ws/secure/structuremap/explainmap
Access Secure
Authentication HTTP Basic Authentication
Http Method POST
Accepts SDMX-ML, SDMX-JSON or SDMX-EDI structure message
Content-Type application/text or application/xml

Response Format JSON object
Response Statuses

200 - Query Ok

400 - Missing Header 'URN' or Unrecognised file input

401 - Unauthorized

404 - Cannot resolve reference to Structure Map specified in URN parameter

HTTP Headers

The following header parameters must be specified:

HTTP Header Purpose Allowed Values
URN

(Mandatory) States the mapping structure to map the input data against.

A valid URN for a Mapping. E.g. urn:sdmx:org.sdmx.infomodel.mapping.StructureMap=MAPPING_TEST:SS_DF(1.0).DF_MAP_TEST

Output

The output JSON object has the following fields:

  • StructureMap - the URN of the structure map used in the mapping
  • SourceStructure - the URN of the DSD (and possibly Dataflow) used in the source and its components
  • TargetStructure - the URN of the DSD (and possibly Dataflow) used in the target and its components
  • Result - a JSON array where each element in the array is a JSON Object representing a single "row" in the mapping.
    • If it is a 'valid' row the object consists of:
      • "Input": the input values as a JSON array of strings
      • "Output": the output values as a JSON array of strings
      • "ValidationErrors": whether there are any validation errors
    • If it is an "invalid" row the object consists of:
      • "Input": the input values as a JSON array of strings
      • "Unmapped" - a JSON object containing:
        • "Output": a JSON array of strings stating the values were produced by the mapping
        • "MissingComponents" - a JSON array of the IDs of the components missing from the output

Example Output

{
    "StructureMap": "urn:sdmx:org.sdmx.infomodel.structuremapping.StructureMap=MAPPING_TEST:SIMPLE_MAPPING_TEST(1.0)",
    "SourceStructure": {
        "DSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=MAPPING_TEST:DSD_SIMPLE_1(1.0)",
        "Dataflow": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=MAPPING_TEST:DF_SIMPLE_1(1.0)",
        "Components": [ "BIS_TOPIC", "REF_AREA", "TIME_PERIOD", "OBS_STATUS", "OBS_PRE_BREAK", "OBS_CONF", "BREAKS", "COVERAGE", "OBS_VALUE" ]
    },
    "TargetStructure": {
        "DSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=MAPPING_TEST:DSD_SIMPLE_2(1.0)",
        "Dataflow": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=MAPPING_TEST:DF_SIMPLE_2(1.0)",
        "Components": [ "INSTRUMENT_TYPE", "REF_AREA", "EER_TYPE", "FREQ", "TIME_PERIOD", "OBS_STATUS", "OBS_PRE_BREAK", "OBS_CONF", "SUPP_INFO_BREAKS", "COVERAGE", "OBS_VALUE" ]
    },
    "Result": [
        {
            "Input": [ "A", "BEL", null, null, null, null, "", "", null ],
            "Unmapped": [
                {
                    "Output": [ "AAAA", null, "A", null, null, null, null, null, null, null, null ],
                    "MissingComponents": [ "REF_AREA" ]
                }
            ]
        },
        {
            "Input": [ "A", "01", "2006", "A", "", "", "", "", "2006" ],
            "Output": [ "AAAA", "BEL", "A", "A", "2006", null, null, null, null, null, "2006" ]
        }
    ]
}

'Explain Plan Series' Web Service

Available from FMR Version 11.10 onwards - the output is the same as the now deprecated 'Explain Plan' Web Service

Given the specified Key and Map, returns a JSON object stating the input, mappings and output for that series key

Entry Point ws/secure/structuremap/explainmapseries
Access Secure
Authentication HTTP Basic Authentication
Http Method POST
Accepts JSON message
Content-Type application/json

Response Format JSON object
Response Statuses

200 - Query Ok

400 - Missing required field in JSON input

401 - Unauthorized

404 - Cannot resolve reference to Structure Map specified in URN parameter

Example Input

{
    "StructureMap": "urn:sdmx:org.sdmx.infomodel.structuremapping.StructureMap=MAPPING_TEST:DF_MAP(1.0)",
    "Series": "A B.C.BAR.2003",
    "MapFrom": "source",
    "ComponentValues": {
        "OBS_STATUS": "5D",
        "OBS_VALUE": "9993"
    }
}

Output

The output JSON object has the fields:

  • StructureMap - the URN of the structure map used in the mapping
  • SourceDSD - the URN of the DSD used in the source
  • TargetDSD - the URN of the DSD used for the target
  • InputRow - a JSON Object where the keys are the IDs of the components used in the input and the values for each key are the values in the row
  • OutputRows - a JSON Array of JSON Objects. Each object is an output mapping which has the fields:
    • Outputs - a JSON Object where the keys are the IDs of the components used in the target output and the values for each key are the mapped values
    • Mappings - a JSON Array of JSON Objects. Each object contains:
      • SourceComponents - an array of the components for the input
      • TargetComponents - an array of the components that are being mapped to
      • Matches - a JSON Object of the input component, input value and the RuleType used. The RuleType can be one of the following values: "Equals", "Copy", "RegEx", "TimeEpoch", "TimePattern"
      • Outputs - a JSON Object of the mapped component and mapped value

Example Output

{
    "StructureMap": "urn:sdmx:org.sdmx.infomodel.structuremapping.StructureMap=MAPPING_TEST:SIMPLE_MAPPING_TEST(1.0)",
    "SourceDSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=MAPPING_TEST:DSD_SIMPLE_1(1.0)",
    "TargetDSD": "urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=MAPPING_TEST:DSD_SIMPLE_2(1.0)",
    "InputRow": {
        "OBS_STATUS": "A",
        "REF_AREA": "01",
        "OBS_VALUE": "2006.01",
        "TIME_PERIOD": "2006-XX",
        "BIS_TOPIC": "A"
    },
    "OutputRows": [
        {
            "Outputs": {
                "AllPeriods": {
                    "INSTRUMENT_TYPE": "AAAA",
                    "EER_TYPE": "A",
                    "REF_AREA": "BEL",
                    "OBS_VALUE": "2006.01"
                }
            },
            "Mappings": [
                {
                    "SourceComponents": [
                        "TIME_PERIOD"
                    ],
                    "TargetComponents": [
                        "TIME_PERIOD"
                    ],
                    "Matches": {
                        "AllPeriods": {
                            "TIME_PERIOD": {
                                "RuleType": "Copy",
                                "Input": "2006-XX"
                            }
                        }
                    },
                    "Outputs": {
                        "AllPeriods": {
                            "TIME_PERIOD": [
                                "2006-XX"
                            ]
                        }
                    }
                },
                {
                    "SourceComponents": [
                        "REF_AREA"
                    ],
                    "TargetComponents": [
                        "REF_AREA"
                    ],
                    "Matches": {
                        "AllPeriods": {
                            "REF_AREA": {
                                "RuleType": "Equals",
                                "Input": "01"
                            }
                        }
                    },
                    "Outputs": {
                        "AllPeriods": {
                            "REF_AREA": [
                                "BEL"
                            ]
                        }
                    }
                },
                {
                    "SourceComponents": [
                        "OBS_VALUE"
                    ],
                    "TargetComponents": [
                        "OBS_VALUE"
                    ],
                    "Matches": {
                        "AllPeriods": {
                            "OBS_VALUE": {
                                "RuleType": "Copy",
                                "Input": "2006.01"
                            }
                        }
                    },
                    "Outputs": {
                        "AllPeriods": {
                            "OBS_VALUE": [
                                "2006.01"
                            ]
                        }
                    }
                },
                {
                    "SourceComponents": [
                        "BIS_TOPIC"
                    ],
                    "TargetComponents": [
                        "EER_TYPE"
                    ],
                    "Matches": {
                        "AllPeriods": {
                            "EER_TYPE": {
                                "RuleType": "Copy",
                                "Input": "A"
                            }
                        }
                    },
                    "Outputs": {
                        "AllPeriods": {
                            "EER_TYPE": [
                                "A"
                            ]
                        }
                    }
                }
            ]
        }
    ]
}

'Explain Plan' Web Service

Deprecated in Fusion Metadata Registry version 11.10: Use 'Explain Plan Series Key' Web Service instead - The new POST service generates the same output as this Web Service

Given the specified Key and Map, returns a JSON object stating the input, mappings and output for that series key

Entry Point ws/secure/structuremap/explainplan
Access Secure
Authentication HTTP Basic Authentication
Http Method GET
Response Format JSON object
Response Statuses

200 - Success

400 - Missing or invalid parameter

401 - Unauthorized

404 - Cannot resolve reference to Structure Map specified in "map" parameter

HTTP Query Parameters

Request Parameter Required Purpose
dsd Yes The URN of the DSD or Dataflow to map against. Depending on whether the source or target DSD / Dataflow is provided, the mapping can be reversed.
map Yes The URN of the Structure Map to use in the mapping
key Yes This is the token that was passed back from the Fusion registry server on the publication request

Output

The output JSON object has the fields:

  • reverseMapping - whether this was a reversed mapping or not
  • input a JSON object stating the dimensions and attributes input. The dimensions will be the same as that passed in as the input parameter "key", but whereas the key only specified the value, this field shows the ID and value relationship.
  • unmapped - a list of any Dimensions that are unmapped
  • mappings - states the input mappings that were mapped as well as any fixed values output
  • output - a JSON object with a field of "dimensions". This field lists the IDs of the output DSD or Dataflow and states what the values would be. If a dimension has a value of "null" it is unmapped. If it has a single value it was successfully mapped. If it has multiple values there this is also a failure as it is ambiguous as to what should be output.

Example Output

{
    "reverseMapping": false,
    "input": {
        "dimensions": {
            "FREQ": "C55BCX",
            "BIS_TOPIC": "C",
            "REF_AREA": "BEL"
        },
        "attributes": {
            "OBS_STATUS": null,
            "OBS_PRE_BREAK": null,
            "OBS_CONF": null,
            "BREAKS": null,
            "COVERAGE": null
        }
    },
    "unmapped": [],
    "mappings": {
        "REF_AREA:BEL": {
            "REF_AREA": [
                "BEL"
            ]
        },
        "OBS_VALUE:*": {
            "EER_TYPE": [
                "FIXED_EER"
            ],
            "COVERAGE": [
                "FIXED_CVRG"
            ]
        }
    },
    "output": {
        "dimensions": {
            "INSTRUMENT_TYPE": [
                null
            ],
            "REF_AREA": [
                "BEL"
            ],
            "EER_TYPE": [
                "FIXED_EER"
            ]
        }
    }
}

Simple Example

The file attached defines a number of Structural Metadata artefacts in the Agency "MAPPING_TEST" that can be loaded into a Registry

  • Test Structure - a zipped file that may be used "as is". It contains 2 Dataflows and 2 DSDs. 2 Concept Schemes, 2 Codelists and 1 Structure Set which maps the Dataflows. All structures are for the agency "MAPPING_TEST".
  • Test Data - this data file should be unzipped before use, to give a small SDMX-ML data file. This file can be loaded against the Dataflows from the Structure file.

Invoking the Explain Map

Once the above Structural Metadata is loaded into your Registry, invoking the "Explain Map" Web Service by performing a POST call (using Curl, Python or POSTMAN) to: 

<your registry>/ws/secure/structuremap/explainmap

This also requires the Header parameter URN to be specified as urn:sdmx:org.sdmx.infomodel.mapping.StructureMap=MAPPING_TEST:SS_DF(1.0).DF_MAP_TEST
The body should be the test file (above).
The resultant output will show that 3 series are mapped and the series "C55BCX:C:BEL" is unmapped. Part of the output will show: 

"mapped": {
        "A:B:BEL": "AAAB:BEL:FIXED_EER",
        "A:C:BEL": "AAAC:BEL:FIXED_EER",
        "XXX:C:BAR": "AACC:BAR:FIXED_EER"
    },
    "unmapped": [
        "C55BCX:C:BEL"
    ]

Invoking the Explain Plan

Now that we know that "C55BCX:C:BEL" is unmapped, we want to call the "explain plan" Web Service, to understand more about the unmapped series "C55BCX:C:BEL". The query needs to specify the Structure Map and specifying the "dsd" to be the Dataflow Reference:

The Dataflow URN can be specified as the "dsd" parameter like so: 

<your registry>/ws/secure/structuremap/explainplan?dsd=urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=MAPPING_TEST:DF_DSD1(1.0)&map=urn:sdmx:org.sdmx.infomodel.mapping.StructureMap=MAPPING_TEST:SS_DF(1.0).DF_MAP_TEST&key=C55BCX:C:BEL
Retrieved from "https://fmrwiki.sdmxcloud.org/index.php?title=Explain_Plan_Web_Services&oldid=7247"