FMR 11.0 release notes

Overview

FMR 11.0 implements the SDMX 3.0 information model and differs from FMR 10 which uses SDMX 2.1.

However, backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes.

Note that FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.

Key new SDMX 3.0 features supported by FMR 11

Microdata modelling

Enhancements to the Data Structure Definition allow microdata to be more easily modelled:

• Multiple measures - SDMX 3.0 no longer has the concept of a primary measure, instead datasets can be modelled simply with 1 or more measures
• Multi-value attributes and measures - alternatively known as 'arrays', each attribute and measure can have an array of values if required
• Multi-lingual text attributes and measures
• Value List – like a Codelist, but with fewer restrictions

Geospatial data modelling

Two specialised forms of codelist have been added for modelling geospatial concepts:

• Geographic Codelist - each code represents a geographic feature described using a 'geofeature set' expression, the syntax for which is defined in the SDMX 3.0 technical specifications
• Geogrid Codelist - for use with gridded geographies, the codelist defines the grid and each code is given specific coordinates within that grid

A new GeospatialInformation type is provided for non-enumerated components. This uses the same 'geofeature set' syntax as for Geographic Codelists and is particualrly useful for geospatial attributes.

Improved structure mapping

Structure mapping allows data to be mapped from one structure (DSD) to another. SDMX 3.0 significantly overhauls the structure mapping model:

• The SDMX 2.1 Structure Set has been removed and replaced by separate Structure Map and Representation Map artefacts to simplify maintenance and promote reuse.
• N-to-N mapping support - A combination of one or more components in the source dataset can be mapped to a combination of one or more components in the target.
• Improved support for mapping time / date values.
• Regex expressions can be used to select values in the source dataset that match a pattern. e.g. ^EUR.*$ maps to EUR
• Mapping validity periods - apply different maps depending on the observation timestamp.

Improved constraint modelling

The constraints model has modified and improved with the introduction of separate data and reference metadata constraints. Note that only data constraints are supported in FMR 11.0 with with the addition of metadata constraints planned for a later release.

• The '%' wildcard can be used when defining Cube Region constraints
• The '+' operator can be used to specify multiple values for a component when defining Series constraints (note that this is not part of the official SDMX 3.0 specification)

Artefact versioning using Semantic Versioning

SDMX 3.0 introduces the option of using Semantic Versioning (https://semver.org) for managing structural metadata artefact versioning and removes the deprecated Final / Non-Final concept. FMR 11.0 supports:

• Three-digit version numbers supporting the Semantic Versiong numbering system
• REST API semantic versioning syntax for querying structures, for instance: 1.3+ any 1 version greater than 1.3 (e.g. 1.5, but not 1.1)

Future releases of FMR 11 will add support for semantic cross references between structures, the working draft version syntax (e.g. 1.0.0-draft) and the option to auto-increment version numbers on change commit.

Other new features in FMR 11

JNDI data source support for Apache Tomcat and RedHat JBoss

JNDI data sources can now be used for FMR's operating database connection when deployed on Tomcat or JBoss web application servers.
Support for Oracle Weblogic is planned for a future FMR 11 release.

Strengthen security by externalising the password used to encrypt database and other passwords

Passwords for database access and other purposes are encrypted using AES-256. In FMR 11, the password for that encryption is randomly generated at install time and stored in the properties file which defaults to fmr.properies. Access to the properties file must therefore be strictly controlled using filesystem access controls. The change makes the password unique to each installation and reduces the risk of security being compromised.

Backward compatibility

• All SDMX 2.1 stuctures can be loaded into a FMR 11 registry, although some such as Structure Maps and Hierarchical Codelists are converted irreversably to the equivalent SDMX 3.0 model structures.
• SDMX 2.1, 2.0 and EDI structure and data formats continue to be supported where possible. Structures such as Representation Maps which only exist in the SDMX 3.0 information model cannot be exported in legacy structure formats. Similarly, data that uses some of the features introduced in SDMX 3.0 for handling microdata cannot be converted to an SDMX 2.1 or older data format.
• The SDMX 2.1 structure REST API syntax and entry point is supported meaning that most systems and processes designed for use with FMR 10 will continue to work with FMR 11.

Breaking changes

FMR 11 incompatible with FMR 10 databases

FMR 11.0 is not compatible with FMR 10 operating databases. Upgrading from FMR 11 therefore requires the software to be installed on a new database schema and existing structural metadata migrated by hand. Upgrading to Version 11 explains how.

Upper case enforced for Maintainable Structure IDs

Maintainable structure IDs are now always upper case in FMR 11 to avoid potential cross-referencing conflicts. The following are valid:

• CL_REF_AREA
• DF_BALANCE_OF_PAYMENTS

While the following are invalid:

• cl_ref_area
• DF_Balance_of_Payments

Structure IDs will automatically be converted to upper case on import. An error will result should attempts be made to import two structures with IDs that differ only in case.

Environment Synchronisation temporarily removed

FMR 11.0 does not provide an Environment Synchronisation tool for moving structures between two Registry installations. The tool is being redesigned to improve usability and performance, and support the SDMX 3.0 information model.

Complete list of features and changes in FMR 11.0

Reference		Summary	Description
FMR11-1	Feature	SDMX 3.0 internal information model	The SDMX 3.0 information model is used internally.
FMR11-2	Feature	Storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map	Support for input, storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map
FMR11-3	Feature	SDMX 3.0 structure query REST API	Structure query REST API to the SDMX 3.0 specification.
FMR11-4	Feature	SDMX 3.0 schema query REST API	Schema query REST API to the SDMX 3.0 specification.
FMR11-5	Feature	SDMX 2.1 compatibility structure query REST API	Structure query REST API to the SDMX 2.1 specification for backward compatibility purposes. The following are not supported: Querying for SDMX 2.1 structure classes that do not exist in the SDMX 3.0 model, e.g. structure sets Querying for SDMX 3.0 structure classes that are not supported by the SDMX 2.1 structure query REST API specification e.g. representation maps
FMR11-6	Feature	SDMX-ML (XML) 3.0 structure format	Structures can be read and written in SDMX 3.0 SDMX-ML.
FMR11-7	Feature	Read and write structures in supported legacy SDMX 2.1, 2.0 and 1.0 structure formats	Structures can be read and written in supported legacy SDMX 2.1, 2.0 and 1.0 formats where the models align sufficiently e.g. simple Codelists, Concept Schemes, Dataflows. The following SDMX 3.0 structures cannot be written in legacy structure formats: SDMX 3.0 structures that have no equivalent in SDMX 2.1 or earlier, e.g. Value Lists, Metadata Provider Schemes SDMX 3.0 structures using features that are not possible to describe in SDMX 2.1 or earlier, e.g. DSDs with multiple measures SDMX 3.0 extended Codelists are materialised when written in legacy formats SDMX 3.0 geospatial Codelists loose their geofeature and geogrid information. Legacy structures that do not exist in the SDMX 3.0 information model are converted on read where possible, including: Structure Sets are converted to the SDMX 3.0 mapping model using Structure Maps and Representation Maps Hierarchical Codelists are converted to the SDMX 3.0 equivalent Hierarchy artefacts
FMR11-8	Feature	Fusion-JSON 3.0 structure format	Fusion-JSON is Metadata Technology's proprietary JSON format for serialising SDMX 3.0 object collections. It is critical for driving the FMR user interfaces, and can also be used for transmission of structural metadata between FMR environments. It differs from SDMX standard formats in that it is able to represent all of the possible structures, including any structures that are not part of the official SDMX standard.
FMR11-9	Feature	SDMX-CSV 3.0 data format (partial)	Read and write data in SDMX 3.0 CSV for the purposes of validation, mapping and conversion to other formats. The majority of the SDMX 3.0 CSV specification is supported with the exception of some advanced features including: multi-lingual attributes multi-lingual measures flat reference metadata attributes Note that no support is planned for complex hierarchical reference metadata in CSV datasets.
FMR11-10	Feature	SDMX 3.0 data validation	Data validation respecting the new SDMX 3.0 model features, principally: multiple measures, array attributes / measures, the enhanced constraints model - % operator, validity of constraining terms, discriminated union extended codelists value lists the new component representation types (e.g. geospatialinformation) Validation of both traditional aggregated time series and SDMX 3.0 micro-datasets is supported.
FMR11-11	Feature	SDMX 3.0 data mapping	Map loaded datasets using the SDMX 3.0 mapping model including: Mapping of micro-datasets Absolute time mapping Excluding: Relative time mapping The mapped resultset can be written out in any of the supported SDMX 3.0 data formats, or a supported legacy data format provided that the selected format is capable representing it.
FMR11-12	Feature	Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load	SDMX 2.1 Structure Sets are automatically converted to the SDMX 3.0 mapping model (Structure Maps and Representation Maps) on load. The conversion is one-way meaning that it is not possible to retrieve the original Structure Set.
FMR11-13	Feature	Read and write data in supported legacy SDMX 2.1 and 2.0 data formats (legacy data conversion)	Datasets can be: Read in a SDMX 3.0 format and converted to a supported legacy format provided that it is capable of representing the data. Read in a supported legacy format and converted to a SDMX 3.0 format. Read in a supported legacy format, mapped to a different structure and written out as a supported SDMX 3.0 data format, or a supported legacy format provided that it is capable of representing the data. SDMX 3.0 aggregated time-series datasets can usually be represented in legacy formats, provided they do not use multi-value attributes or measures which are features specific to SDMX 3.0. SDMX 3.0 micro-datasets cannot be represented in legacy data formats. Datasets loaded in legacy formats can generally be converted to SDMX 3.0 data formats without issue.
FMR11-14	Feature	Automatic conversion of SDMX legacy Hierarchical Codelists to the SDMX 3.0 Hierarchy model on load	SDMX 2.1 Hierarchical Codelists are automatically converted to the SDMX 3.0 Hierarchy model on load. The conversion is two-way meaning that it is possible to retrieve the original Hierarchical Codelist once the load is complete. However, the conversion is lossy such that HCL's are ungrouped on the conversion to Hierarchies.
FMR11-15	Feature	EDI legacy structure format	Read and write structures in pre-SDMX 3.0 EDI format where possible.
FMR11-16	Feature	SDMX-ML (XML) 2.1, 2.0 and 1.0 legacy structure formats	Read and write structural metadata in pre-SDMX 3.0 XML structure formats where possible.
FMR11-18	Feature	SDMX-ML (XML) 2.1 and 2.0 legacy data formats	Read and write data in the following legacy SDMX-ML formats where possible: SDMX 2.1 Structure Specific SDMX 2.1 Generic SDMX 2.0 Compact SDMX 2.0 Generic FMR11's data conversion rules define when datasets can be read and written in legacy formats.
FMR11-21	Feature	SDMX-JSON 2.1 legacy structure format	Read and write structural metadata in pre-SDMX 3.0 JSON structure formats where possible.
FMR11-22	Feature	Read and write structures in supported SDMX 3.0 structure formats	Structures can be read and written in supported SDMX 3.0 structure formats.
FMR11-23	Feature	Read and write data in supported SDMX 3.0 data formats	Read and write data in supported SDMX 3.0 data formats
FMR11-28	Feature	UI - support annotations on Representation Maps	Structure mapping designers can add explanatory notes to each Representation Map which are stored as SDMX annotations.
FMR11-50	Feature	Series constraint expression using the + operator	Use the '+' operator to specify multiple values for series constraint values to simplify maintenance and reduce the verbosity. Example SDMX-ML <str:Key> <com:KeyValue id="REF_AREA"> <com:Value>DE+FR+GB</com:Value> </com:KeyValue> <com:KeyValue id="INDICATOR"> <com:Value>ABC_123</com:Value> </com:KeyValue> </str:Key> The above evaluates to the following series keys: DE:ABC_123 FR:ABC_123 GB:ABC_123
FMR11-51	Feature	Semantic versioning - three-digit version numbers (e.g. 1.3.2)	Structures can be versioned using the three-digit semantic versioning syntax.
FMR11-52	Feature	Semantic versioning - REST API structure query using semantic versioning syntax	REST API structure queries support the semantic versioning selection syntax using the '+' operator. e.g. + latest stable version 1.3+ any 1 version greater than 1.3 (e.g. 1.5, but not 1.1) 1.0.0,2.1.7 either version 1.0.0 or 2.1.7
FMR11-62	Feature	JNDI data sources - Apache Tomcat and RedHat JBoss support	JNDI data source has been added to the list of FMR database connection options in addition to Oracle, MySQL and SQL Server. Apache Tomcat and RedHat JBoss are supported. Support for Oracle Weblogic is planned for a future release.
FMR11-60	Change	Change to the structure of the operating database	The structure of the operating database has changed in FMR 11 making it incompatible with version 10. An existing FMR 10 installation cannot be directly upgraded to FMR 11, and an FMR 11 database cannot be used with version 10 of the software. To upgrade from 10 to 11: Create a new FMR 11 installation using a separate database schema Export the structural metadata from FMR 10 to file using the Fusion-JSON format Import the structural metadata into FMR11 which will perform an automatic conversion to the SDMX 3.0 model
FMR11-61	Change	Environment synchronisation function temporarily removed	The Environment Synchronisation function has been temporarily removed in FMR 11.0 but will be replaced in a future 11 released with a new design.
FMR11-69	Change	Upper case enforced for Maintainable Structure IDs	Maintainable structure IDs must be upper case - lower case and mixed case IDs are not supported. IDs are converted to upper case on structure import. Attempting to import structures with IDs that differ only in case will result in an error.