Difference between revisions of "FMR 11.0 release notes"

From FMR Knowledge Base
Jump to navigation Jump to search
(Key new SDMX 3.0 features supported by FMR 11)
(Key new SDMX 3.0 features supported by FMR 11)
Line 25: Line 25:
 
* Regex expressions can be used to select values in the source dataset that match a pattern. e.g. ^EUR.*$ maps to EUR
 
* Regex expressions can be used to select values in the source dataset that match a pattern. e.g. ^EUR.*$ maps to EUR
 
* Mapping validity periods - apply different maps depending on the observation timestamp.
 
* Mapping validity periods - apply different maps depending on the observation timestamp.
 +
 +
====Improved constraint modelling====
 +
The constraints model has modified and improved with the introduction of separate data and reference metadata constraints. Note that only data constraints are supported in FMR 11.0 with with the addition of metadata constraints planned for a later release.
 +
* The '%' wildcard can be used when defining Cube Region constraints
 +
* The '+' operator can be used to specify multiple values for a component when defining Series constraints (note that this is not part of the official SDMX 3.0 specification)
  
 
=Backward compatibility=
 
=Backward compatibility=

Revision as of 05:27, 28 December 2021

FMR 11.0 implements the SDMX 3.0 information model and differs fundamentally in that respect from FMR 10 which is designed around SDMX 2.1.

Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes. However, FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.

Key new SDMX 3.0 features supported by FMR 11

Microdata modelling

Enhancements to the Data Structure Definition allow microdata to be more easily modelled:

  • Multiple measures - SDMX 3.0 no longer has the concept of a primary measure, instead datasets can be modelled simply with 1 or more measures
  • Multi-value attributes and measures - alternatively known as 'arrays', each attribute and measure can have an array of values if required
  • Multi-lingual text attributes and measures
  • Value List – like a Codelist, but with fewer restrictions

Geospatial data modelling

Two specialised forms of codelist have been added for modelling geospatial concepts:

  • Geographic Codelist - each code represents a geographic feature described using a 'geofeature set' expression, the syntax for which is defined in the SDMX 3.0 technical specifications
  • Geogrid Codelist - for use with gridded geographies, the codelist defines the grid and each code is given specific coordinates within that grid

A new GeospatialInformation type is provided for non-enumerated components. This uses the same 'geofeature set' syntax as for Geographic Codelists and is particualrly useful for geospatial attributes.

Improved structure mapping

Structure mapping allows data to be mapped from one structure (DSD) to another. SDMX 3.0 significantly overhauls the structure mapping model:

  • The SDMX 2.1 Structure Set has been removed and replaced by separate Structure Map and Representation Map artefacts to simplify maintenance and promote reuse.
  • N-to-N mapping support - A combination of one or more components in the source dataset can be mapped to a combination of one or more components in the target.
  • Improved support for mapping time / date values.
  • Regex expressions can be used to select values in the source dataset that match a pattern. e.g. ^EUR.*$ maps to EUR
  • Mapping validity periods - apply different maps depending on the observation timestamp.

Improved constraint modelling

The constraints model has modified and improved with the introduction of separate data and reference metadata constraints. Note that only data constraints are supported in FMR 11.0 with with the addition of metadata constraints planned for a later release.

  • The '%' wildcard can be used when defining Cube Region constraints
  • The '+' operator can be used to specify multiple values for a component when defining Series constraints (note that this is not part of the official SDMX 3.0 specification)

Backward compatibility

  • All SDMX 2.1 stuctures can be loaded into a FMR 11 registry, although some such as Structure Maps and Hierarchical Codelists are converted irreversably to the equivalent SDMX 3.0 model structures.
  • SDMX 2.1, 2.0 and EDI structure and data formats continue to be supported where possible. Structures such as Representation Maps which only exist in the SDMX 3.0 information model cannot be exported in legacy structure formats. Similarly, data that uses some of the features introduced in SDMX 3.0 for handling microdata cannot be converted to an SDMX 2.1 or older data format.
  • The SDMX 2.1 structure REST API syntax and entry point is supported meaning that most systems and processes designed for use with FMR 10 will continue to work with FMR 11.

Breaking change - FMR 11 incompatible with FMR 10 databases

FMR 11.0 is not compatible with FMR 10 operating databases. Upgrading from FMR 11 therefore requires the software to be installed on a new database schema and existing structural metadata migrated by hand. Upgrading to Version 11 explains how.

Environment Synchronisation temporarily removed

FMR 11.0 does not provide an Environment Synchronisation tool for moving structures between two Registry installations. The tool is being redesigned to improve usability and performance, and support the SDMX 3.0 information model.

Features and changes

Reference


Summary

Description

FMR11-1

New Feature

SDMX 3.0 internal information model

The SDMX 3.0 information model is used internally.

FMR11-2

New Feature

Storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map

Support for input, storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map

FMR11-3

New Feature

SDMX 3.0 structure query REST API

Structure query REST API to the SDMX 3.0 specification.

FMR11-4

New Feature

SDMX 3.0 schema query REST API

Schema query REST API to the SDMX 3.0 specification.

FMR11-5

New Feature

SDMX 2.1 compatibility structure query REST API

Structure query REST API to the SDMX 2.1 specification for backward compatibility purposes.

The following are not supported:

  • Querying for SDMX 2.1 structure classes that do not exist in the SDMX 3.0 model, e.g. structure sets
  • Querying for SDMX 3.0 structure classes that are not supported by the SDMX 2.1 structure query REST API specification e.g. representation maps

FMR11-6

New Feature

SDMX-ML (XML) 3.0 structure format

Structures can be read and written in SDMX 3.0 SDMX-ML.

FMR11-7

New Feature

Read and write structures in supported legacy SDMX 2.1, 2.0 and 1.0 structure formats

Structures can be read and written in supported legacy SDMX 2.1, 2.0 and 1.0 formats where the models align sufficiently e.g. simple Codelists, Concept Schemes, Dataflows.

The following SDMX 3.0 structures cannot be written in legacy structure formats:


  • SDMX 3.0 structures that have no equivalent in SDMX 2.1 or earlier, e.g. Value Lists, Metadata Provider Schemes
  • SDMX 3.0 structures using features that are not possible to describe in SDMX 2.1 or earlier, e.g. DSDs with multiple measures

SDMX 3.0 extended Codelists are materialised when written in legacy formats

SDMX 3.0 geospatial Codelists loose their geofeature and geogrid information.

Legacy structures that do not exist in the SDMX 3.0 information model are converted on read where possible, including:

  • Structure Sets are converted to the SDMX 3.0 mapping model using Structure Maps and Representation Maps
  • Hierarchical Codelists are converted to the SDMX 3.0 equivalent Hierarchy artefacts

FMR11-8

New Feature

Fusion-JSON 3.0 structure format

Fusion-JSON is Metadata Technology's proprietary JSON format for serialising SDMX 3.0 object collections. It is critical for driving the FMR user interfaces, and can also be used for transmission of structural metadata between FMR environments. It differs from SDMX standard formats in that it is able to represent all of the possible structures, including any structures that are not part of the official SDMX standard.

FMR11-9

New Feature

SDMX-CSV 3.0 data format (partial)

Read and write data in SDMX 3.0 CSV for the purposes of validation, mapping and conversion to other formats.


The majority of the SDMX 3.0 CSV specification is supported with the exception of some advanced features including:


  • multi-lingual attributes
  • multi-lingual measures
  • hierarchical reference metadata

FMR11-10

New Feature

SDMX 3.0 data validation

Data validation respecting the new SDMX 3.0 model features, principally:

  • multiple measures, array attributes / measures,
  • the enhanced constraints model - % operator, validity of constraining terms, discriminated union
  • extended codelists
  • value lists
  • the new component representation types (e.g. geospatialinformation)


Validation of both traditional aggregated time series and SDMX 3.0 micro-datasets is supported.

FMR11-11

New Feature

SDMX 3.0 data mapping

Map loaded datasets using the SDMX 3.0 mapping model including:


  • Mapping of micro-datasets
  • Absolute time mapping

Excluding:

  • Relative time mapping

The mapped resultset can be written out in any of the supported SDMX 3.0 data formats, or a supported legacy data format provided that the selected format is capable representing it.

FMR11-12

New Feature

Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load

SDMX 2.1 Structure Sets are automatically converted to the SDMX 3.0 mapping model (Structure Maps and Representation Maps) on load.


The conversion is one-way meaning that it is not possible to retrieve the original Structure Set.

FMR11-13

New Feature

Read and write data in supported legacy SDMX 2.1 and 2.0 data formats (legacy data conversion)

Datasets can be:

  • Read in a SDMX 3.0 format and converted to a supported legacy format provided that it is capable of representing the data.
  • Read in a supported legacy format and converted to a SDMX 3.0 format.
  • Read in a supported legacy format, mapped to a different structure and written out as a supported SDMX 3.0 data format, or a supported legacy format provided that it is capable of representing the data.

SDMX 3.0 aggregated time-series datasets can usually be represented in legacy formats, provided they do not use multi-value attributes or measures which are features specific to SDMX 3.0.

SDMX 3.0 micro-datasets cannot be represented in legacy data formats.

Datasets loaded in legacy formats can generally be converted to SDMX 3.0 data formats without issue.

FMR11-14

New Feature

Automatic conversion of SDMX legacy Hierarchical Codelists to the SDMX 3.0 Hierarchy model on load

SDMX 2.1 Hierarchical Codelists are automatically converted to the SDMX 3.0 Hierarchy model on load.


The conversion is two-way meaning that it is possible to retrieve the original Hierarchical Codelist once the load is complete. However, the conversion is lossy such that HCL's are ungrouped on the conversion to Hierarchies.

FMR11-15

New Feature

EDI legacy structure format

Read and write structures in pre-SDMX 3.0 EDI format where possible.

FMR11-16

New Feature

SDMX-ML (XML) 2.1, 2.0 and 1.0 legacy structure formats

Read and write structural metadata in pre-SDMX 3.0 XML structure formats where possible.

FMR11-18

New Feature

SDMX-ML (XML) 2.1 and 2.0 legacy data formats

Read and write data in the following legacy SDMX-ML formats where possible:

  • SDMX 2.1 Structure Specific
  • SDMX 2.1 Generic
  • SDMX 2.0 Compact
  • SDMX 2.0 Generic

FMR11's data conversion rules define when datasets can be read and written in legacy formats.

FMR11-21

New Feature

SDMX-JSON 2.1 legacy structure format

Read and write structural metadata in pre-SDMX 3.0 JSON structure formats where possible.

FMR11-22

New Feature

Read and write structures in supported SDMX 3.0 structure formats

Structures can be read and written in supported SDMX 3.0 structure formats.

FMR11-23

New Feature

Read and write data in supported SDMX 3.0 data formats

Read and write data in supported SDMX 3.0 data formats

FMR11-28

New Feature

UI - support annotations on Representation Maps

GitHub issue cross reference: [https://github.com/MEDIT-Org/fmr/issues/48|https://github.com/MEDIT-Org/fmr/issues/48|smart-link]

FMR11-50

New Feature

Series constraint expression using the + operator

Use the '+' operator to specify multiple values for series constraint values to simplify maintenance and reduce the verbosity.

Example SDMX-ML

<str:Key>
  <com:KeyValue id="REF_AREA">
     <com:Value>DE+FR+GB</com:Value>
  </com:KeyValue>
  <com:KeyValue id="INDICATOR">
     <com:Value>ABC_123</com:Value>
  </com:KeyValue>
</str:Key>

The above evaluates to the following series keys:

DE:ABC_123
FR:ABC_123
GB:ABC_123

FMR11-51

New Feature

Semantic versioning - three-digit version numbers (e.g. 1.3.2)

Structures can be versioned using the three-digit semantic versioning syntax.

FMR11-52

New Feature

Semantic versioning - REST API structure query using semantic versioning syntax

REST API structure queries support the semantic versioning selection syntax using the '+' operator.

e.g.

+ latest stable version

1.3+ any 1 version greater than 1.3 (e.g. 1.5, but not 1.1)

1.0.0,2.1.7 either version 1.0.0 or 2.1.7

FMR11-62

New Feature

JNDI data sources - Apache Tomcat and RedHat JBoss support

JNDI data source has been added to the list of FMR database connection options in addition to Oracle, MySQL and SQL Server. Apache Tomcat and RedHat JBoss are supported. Support for Oracle Weblogic is planned for a future release.

FMR11-60

Change

Change to the structure of the operating database

The structure of the operating database has changed in FMR 11 making it incompatible with version 10. An existing FMR 10 installation cannot be directly upgraded to FMR 11, and an FMR 11 database cannot be used with version 10 of the software.

To upgrade from 10 to 11:

  • Create a new FMR 11 installation using a separate database schema
  • Export the structural metadata from FMR 10 to file using the Fusion-JSON format
  • Import the structural metadata into FMR11 which will perform an automatic conversion to the SDMX 3.0 model

FMR11-61

Change

Environment synchronisation function temporarily removed

The Environment Synchronisation function has been temporarily removed in FMR 11.0 but will be replaced in a future 11 released with a new design.