Check Database Consistency Web Service

Overview

The Fusion Metadata Registry (version 11), stores maintainable structures in a number of forms within its database. It is extremely unlikely that these formats become inconsistent, but a mechanism exists to check that the various formats are consistent. This mechanism is a simple Web Service that can be invoked by any user with administration privileges. This Web Service is available in FMR version 11.19.6 and later.

Detail

The Fusion Metadata Registry stores structure metadata in the following tables:

• registry_maintainable_ser: this table stores the Java serialised form of each individual metadata structure. There serialised objects are loaded on application startup, since it is faster than constructing a new Java object from the XML.
• registry_maintainable_sdmx: this table stores the SDMX-ML form of the individual metadata structure. This table is a backup for the "serialised" table. If an structure cannot be deserialsed, then the structure is obtained from this table.
• tx_backup: this table stores the SDMX-ML form of structures, grouped together by the transaction they were submitted within rather than storing each structure individually.

When a new structure is submitted, or an existing structure is modified, each of the above 3 tables will be updated.

Check Database Consistency Web Service

Entry Point	/ws/secure/structure/databaseconsistency
Access	Secure
Http Method	GET
Response Format	JSON
Response Statuses	200 - Success 400 - Unauthorized

Note: Because the service has to communicate with the database, it may take a while to run. The time taken depends upon the speed of connection to the database and the number of structural metadata within your Registry. If you wish to see the progress of the Web Service inspect the Fusion Registry log. The process will write a number of log statements (at INFO level) which will state the number of the current item being checked, along with the total number of items to process. An example of a log entry showing that the process is at structure number 226 out of a total of 2,053 items is:

   io.sdmx.core.fmr.manager.report.DatabaseConsistencyChecker - Evaluating (226/2053)

Example Output

The Web Service produces JSON output stating the following:

• is the database consistent
• the number of maintainables that were checked
• a JSON Array listing the URNs of Maintainable Structures that differ in the table 'registry_maintainable_sdmx'
• a JSON Array listing the URNs of Maintainable Structures that differ in the table 'tx_backup'

An example of the full output:

{
  "Consistency": false,
  "MaintainablesChecked": 2053,
  "DifferingSdmx": [
    "[]"
  ],
  "DifferingTxn": [
    "[urn:sdmx:org.sdmx.infomodel.codelist.Codelist=SDMX:AAA(1.0)]"
  ]
}

This shows that the database is not consistent, and that 2,053 items were checked. Since the field 'DifferingSdmx' has an empty JSON array, the items in the table 'registry_maintainable_sdmx' do not differ to the metadata in memory. The field 'DifferingTxn" has a JSON array containing a single URN for the codelist SDMX:AAA(1.0). This means that the codelist stored in the table 'tx_backup' is not consistent with the codelist held in memory.

To determine what the differences are, please refer to the log file of the Registry. An example of output stating the difference is:

 WARN qtp258754732-94 io.sdmx.core.fmr.manager.report.DatabaseConsistencyChecker - Difference exists for : urn:sdmx:org.sdmx.infomodel.codelist.Codelist=SDMX:AAA(1.0)
 WARN qtp258754732-94 io.sdmx.core.fmr.manager.report.DatabaseConsistencyChecker - [EDIT,urn:sdmx:org.sdmx.infomodel.codelist.Codelist=SDMX:AAA(1.0),Name,AAA,AAA-Updated]

This shows that for this codelist, there is a difference ('EDIT') for the property 'Name' and that in memory the codelist name is 'AAA' but in the database it is stored as 'AAA-Updated'.

Examples

An example of the consistency checker being invoked:

https://example.registry.com/ws/secure/structure/databaseconsistency

Known Issues

If your system has codelists which use inheritance to inherit from other codelists, you may receive reports stating that differences exists. This is a known issue and will be addressed in a later release.