Difference between revisions of "FMR 11.0 release notes"
(→Complete list of features and changes in FMR 11.0) |
(→Breaking change - FMR 11 incompatible with FMR 10 databases) |
||
Line 47: | Line 47: | ||
* The SDMX 2.1 structure REST API syntax and entry point is supported meaning that most systems and processes designed for use with FMR 10 will continue to work with FMR 11. | * The SDMX 2.1 structure REST API syntax and entry point is supported meaning that most systems and processes designed for use with FMR 10 will continue to work with FMR 11. | ||
− | =Breaking | + | =Breaking changes= |
+ | ====FMR 11 incompatible with FMR 10 databases==== | ||
FMR 11.0 is not compatible with FMR 10 operating databases. Upgrading from FMR 11 therefore requires the software to be installed on a new database schema and existing structural metadata migrated by hand. [[Upgrading to Version 11]] explains how. | FMR 11.0 is not compatible with FMR 10 operating databases. Upgrading from FMR 11 therefore requires the software to be installed on a new database schema and existing structural metadata migrated by hand. [[Upgrading to Version 11]] explains how. | ||
Revision as of 02:40, 29 December 2021
FMR 11.0 implements the SDMX 3.0 information model and differs fundamentally in that respect from FMR 10 which is designed around SDMX 2.1.
Backward compatibility with SDMX 2.1 and FMR 10 has been preserved where possible allowing upgrade to FMR 11 with few changes to operating procedures and intergated systems / processes. However, FMR 11 is not compatible with existing FMR 10 databases requiring a fresh install and export / import of the structural metadata content.
Contents
Key new SDMX 3.0 features supported by FMR 11
Microdata modelling
Enhancements to the Data Structure Definition allow microdata to be more easily modelled:
- Multiple measures - SDMX 3.0 no longer has the concept of a primary measure, instead datasets can be modelled simply with 1 or more measures
- Multi-value attributes and measures - alternatively known as 'arrays', each attribute and measure can have an array of values if required
- Multi-lingual text attributes and measures
- Value List – like a Codelist, but with fewer restrictions
Geospatial data modelling
Two specialised forms of codelist have been added for modelling geospatial concepts:
- Geographic Codelist - each code represents a geographic feature described using a 'geofeature set' expression, the syntax for which is defined in the SDMX 3.0 technical specifications
- Geogrid Codelist - for use with gridded geographies, the codelist defines the grid and each code is given specific coordinates within that grid
A new GeospatialInformation
type is provided for non-enumerated components. This uses the same 'geofeature set' syntax as for Geographic Codelists and is particualrly useful for geospatial attributes.
Improved structure mapping
Structure mapping allows data to be mapped from one structure (DSD) to another. SDMX 3.0 significantly overhauls the structure mapping model:
- The SDMX 2.1 Structure Set has been removed and replaced by separate Structure Map and Representation Map artefacts to simplify maintenance and promote reuse.
- N-to-N mapping support - A combination of one or more components in the source dataset can be mapped to a combination of one or more components in the target.
- Improved support for mapping time / date values.
- Regex expressions can be used to select values in the source dataset that match a pattern. e.g. ^EUR.*$ maps to EUR
- Mapping validity periods - apply different maps depending on the observation timestamp.
Improved constraint modelling
The constraints model has modified and improved with the introduction of separate data and reference metadata constraints. Note that only data constraints are supported in FMR 11.0 with with the addition of metadata constraints planned for a later release.
- The '%' wildcard can be used when defining Cube Region constraints
- The '+' operator can be used to specify multiple values for a component when defining Series constraints (note that this is not part of the official SDMX 3.0 specification)
Artefact versioning using Semantic Versioning
SDMX 3.0 introduces the option of using Semantic Versioning (https://semver.org) for managing structural metadata artefact versioning and removes the deprecated Final / Non-Final concept. FMR 11.0 supports:
- Three-digit version numbers supporting the Semantic Versiong numbering system
- REST API semantic versioning syntax for querying structures, for instance:
1.3+
any 1 version greater than 1.3 (e.g. 1.5, but not 1.1)
Future releases of FMR 11 will add support for semantic cross references between structures, the working draft version syntax (e.g. 1.0.0-draft) and the option to auto-increment version numbers on change commit.
Other new features in FMR 11
JNDI data source support for Apache Tomcat and RedHat JBoss
JNDI data sources can now be used for FMR's operating database connection when deployed on Tomcat or JBoss web application servers.
Support for Oracle Weblogic is planned for a future FMR 11 release.
Backward compatibility
- All SDMX 2.1 stuctures can be loaded into a FMR 11 registry, although some such as Structure Maps and Hierarchical Codelists are converted irreversably to the equivalent SDMX 3.0 model structures.
- SDMX 2.1, 2.0 and EDI structure and data formats continue to be supported where possible. Structures such as Representation Maps which only exist in the SDMX 3.0 information model cannot be exported in legacy structure formats. Similarly, data that uses some of the features introduced in SDMX 3.0 for handling microdata cannot be converted to an SDMX 2.1 or older data format.
- The SDMX 2.1 structure REST API syntax and entry point is supported meaning that most systems and processes designed for use with FMR 10 will continue to work with FMR 11.
Breaking changes
FMR 11 incompatible with FMR 10 databases
FMR 11.0 is not compatible with FMR 10 operating databases. Upgrading from FMR 11 therefore requires the software to be installed on a new database schema and existing structural metadata migrated by hand. Upgrading to Version 11 explains how.
Environment Synchronisation temporarily removed
FMR 11.0 does not provide an Environment Synchronisation tool for moving structures between two Registry installations. The tool is being redesigned to improve usability and performance, and support the SDMX 3.0 information model.
Complete list of features and changes in FMR 11.0
Reference |
|
Summary |
Description |
FMR11-1 |
Feature |
SDMX 3.0 internal information model |
The SDMX 3.0 information model is used internally. |
FMR11-2 |
Feature |
Storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map |
Support for input, storage and retrieval of all SDMX 3.0 structures with the exception of Metadata Constraints, Concept Scheme Map, Category Scheme Map |
FMR11-3 |
Feature |
SDMX 3.0 structure query REST API |
Structure query REST API to the SDMX 3.0 specification. |
FMR11-4 |
Feature |
SDMX 3.0 schema query REST API |
Schema query REST API to the SDMX 3.0 specification. |
FMR11-5 |
Feature |
SDMX 2.1 compatibility structure query REST API |
Structure query REST API to the SDMX 2.1 specification for backward compatibility purposes. The following are not supported:
|
FMR11-6 |
Feature |
SDMX-ML (XML) 3.0 structure format |
Structures can be read and written in SDMX 3.0 SDMX-ML. |
FMR11-7 |
Feature |
Read and write structures in supported legacy SDMX 2.1, 2.0 and 1.0 structure formats |
Structures can be read and written in supported legacy SDMX 2.1, 2.0 and 1.0 formats where the models align sufficiently e.g. simple Codelists, Concept Schemes, Dataflows. The following SDMX 3.0 structures cannot be written in legacy structure formats:
SDMX 3.0 extended Codelists are materialised when written in legacy formats SDMX 3.0 geospatial Codelists loose their geofeature and geogrid information. Legacy structures that do not exist in the SDMX 3.0 information model are converted on read where possible, including:
|
FMR11-8 |
Feature |
Fusion-JSON 3.0 structure format |
Fusion-JSON is Metadata Technology's proprietary JSON format for serialising SDMX 3.0 object collections. It is critical for driving the FMR user interfaces, and can also be used for transmission of structural metadata between FMR environments. It differs from SDMX standard formats in that it is able to represent all of the possible structures, including any structures that are not part of the official SDMX standard. |
FMR11-9 |
Feature |
SDMX-CSV 3.0 data format (partial) |
Read and write data in SDMX 3.0 CSV for the purposes of validation, mapping and conversion to other formats.
The majority of the SDMX 3.0 CSV specification is supported with the exception of some advanced features including:
Note that no support is planned for complex hierarchical reference metadata in CSV datasets. |
FMR11-10 |
Feature |
SDMX 3.0 data validation |
Data validation respecting the new SDMX 3.0 model features, principally:
|
FMR11-11 |
Feature |
SDMX 3.0 data mapping |
Map loaded datasets using the SDMX 3.0 mapping model including:
Excluding:
The mapped resultset can be written out in any of the supported SDMX 3.0 data formats, or a supported legacy data format provided that the selected format is capable representing it. |
FMR11-12 |
Feature |
Automatic conversion of SDMX 2.1 structure sets to the SDMX 3.0 mapping model on load |
SDMX 2.1 Structure Sets are automatically converted to the SDMX 3.0 mapping model (Structure Maps and Representation Maps) on load.
The conversion is one-way meaning that it is not possible to retrieve the original Structure Set. |
FMR11-13 |
Feature |
Read and write data in supported legacy SDMX 2.1 and 2.0 data formats (legacy data conversion) |
Datasets can be:
SDMX 3.0 aggregated time-series datasets can usually be represented in legacy formats, provided they do not use multi-value attributes or measures which are features specific to SDMX 3.0. SDMX 3.0 micro-datasets cannot be represented in legacy data formats. Datasets loaded in legacy formats can generally be converted to SDMX 3.0 data formats without issue. |
FMR11-14 |
Feature |
Automatic conversion of SDMX legacy Hierarchical Codelists to the SDMX 3.0 Hierarchy model on load |
SDMX 2.1 Hierarchical Codelists are automatically converted to the SDMX 3.0 Hierarchy model on load.
The conversion is two-way meaning that it is possible to retrieve the original Hierarchical Codelist once the load is complete. However, the conversion is lossy such that HCL's are ungrouped on the conversion to Hierarchies. |
FMR11-15 |
Feature |
EDI legacy structure format |
Read and write structures in pre-SDMX 3.0 EDI format where possible. |
FMR11-16 |
Feature |
SDMX-ML (XML) 2.1, 2.0 and 1.0 legacy structure formats |
Read and write structural metadata in pre-SDMX 3.0 XML structure formats where possible. |
FMR11-18 |
Feature |
SDMX-ML (XML) 2.1 and 2.0 legacy data formats |
Read and write data in the following legacy SDMX-ML formats where possible:
FMR11's data conversion rules define when datasets can be read and written in legacy formats. |
FMR11-21 |
Feature |
SDMX-JSON 2.1 legacy structure format |
Read and write structural metadata in pre-SDMX 3.0 JSON structure formats where possible. |
FMR11-22 |
Feature |
Read and write structures in supported SDMX 3.0 structure formats |
Structures can be read and written in supported SDMX 3.0 structure formats. |
FMR11-23 |
Feature |
Read and write data in supported SDMX 3.0 data formats |
Read and write data in supported SDMX 3.0 data formats |
FMR11-28 |
Feature |
UI - support annotations on Representation Maps |
Structure mapping designers can add explanatory notes to each Representation Map which are stored as SDMX annotations. |
FMR11-50 |
Feature |
Series constraint expression using the + operator |
Use the '+' operator to specify multiple values for series constraint values to simplify maintenance and reduce the verbosity. Example SDMX-ML <str:Key> <com:KeyValue id="REF_AREA"> <com:Value>DE+FR+GB</com:Value> </com:KeyValue> <com:KeyValue id="INDICATOR"> <com:Value>ABC_123</com:Value> </com:KeyValue> </str:Key> The above evaluates to the following series keys: DE:ABC_123 FR:ABC_123 GB:ABC_123 |
FMR11-51 |
Feature |
Semantic versioning - three-digit version numbers (e.g. 1.3.2) |
Structures can be versioned using the three-digit semantic versioning syntax. |
FMR11-52 |
Feature |
Semantic versioning - REST API structure query using semantic versioning syntax |
REST API structure queries support the semantic versioning selection syntax using the '+' operator. e.g. + latest stable version 1.3+ any 1 version greater than 1.3 (e.g. 1.5, but not 1.1) 1.0.0,2.1.7 either version 1.0.0 or 2.1.7 |
FMR11-62 |
Feature |
JNDI data sources - Apache Tomcat and RedHat JBoss support |
JNDI data source has been added to the list of FMR database connection options in addition to Oracle, MySQL and SQL Server. Apache Tomcat and RedHat JBoss are supported. Support for Oracle Weblogic is planned for a future release. |
FMR11-60 |
Change |
Change to the structure of the operating database |
The structure of the operating database has changed in FMR 11 making it incompatible with version 10. An existing FMR 10 installation cannot be directly upgraded to FMR 11, and an FMR 11 database cannot be used with version 10 of the software. To upgrade from 10 to 11:
|
FMR11-61 |
Change |
Environment synchronisation function temporarily removed |
The Environment Synchronisation function has been temporarily removed in FMR 11.0 but will be replaced in a future 11 released with a new design. |