Difference between revisions of "DRAFT SDMX-CORE"

From FMR Knowledge Base
Jump to navigation Jump to search
(High Level Design)
(Using Spring/SDMX.IO)
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Architectire =
+
= Architecture =
 
== Modules ==
 
== Modules ==
 
The sdmx-io modules are as follows
 
The sdmx-io modules are as follows
Line 6: Line 6:
  
 
  '''fusion-api'''  - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception
 
  '''fusion-api'''  - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception
  '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects
+
  '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects
  '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules
+
  '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules
  '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies
+
  '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies
  
 
===Implementation Modules===
 
===Implementation Modules===
Line 19: Line 19:
  
 
===Format Modules===
 
===Format Modules===
These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data
+
These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data
  
 
  '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format
 
  '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format
Line 30: Line 30:
 
  '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option
 
  '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option
 
  '''fusion-audit''' - audit events and centralised error handling
 
  '''fusion-audit''' - audit events and centralised error handling
  '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka
+
  '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka
  '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition
+
  '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition
  '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks
+
  '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks
 
  '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service
 
  '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service
 
  '''fusion-sdmx-search''' - Lucene search engine for structures
 
  '''fusion-sdmx-search''' - Lucene search engine for structures
 
  '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)
 
  '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)
  '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin
+
  '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin
  
 
===Utility Modules===
 
===Utility Modules===
Line 44: Line 44:
 
  '''fusion-utils-http''' - calling HTTP web services
 
  '''fusion-utils-http''' - calling HTTP web services
 
  '''fusion-utils-json''' - JSON related utility classes
 
  '''fusion-utils-json''' - JSON related utility classes
  '''fusion-utils-mail''' - email utiliity classes
+
  '''fusion-utils-mail''' - email utility classes
 
  '''fusion-utils-mongodb'''  - implementation code to use MongoDB a means to store Objects
 
  '''fusion-utils-mongodb'''  - implementation code to use MongoDB a means to store Objects
  '''fusion-utils-sdmx''' - SDMX Specific utilty classes
+
  '''fusion-utils-sdmx''' - SDMX Specific utility classes
 
  '''fusion-utils-spring''' - spring related utility classes
 
  '''fusion-utils-spring''' - spring related utility classes
 
  '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes
 
  '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes
  
 
== Java Packages ==
 
== Java Packages ==
Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function.  
+
Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function.  
  
 
The generic packages are as follows.
 
The generic packages are as follows.
  
  '''api''' - contains interfaces relevent to the module  
+
  '''api''' - contains interfaces relevant to the module  
 +
'''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later)
 
  '''constant''' - contains Java enums and java classes that contain public static constants
 
  '''constant''' - contains Java enums and java classes that contain public static constants
 
  '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine  
 
  '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine  
  '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
+
  '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
 
  '''filter'''  - REST web service filters (defined in web.xml) - filters are called before the web service
 
  '''filter'''  - REST web service filters (defined in web.xml) - filters are called before the web service
  '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
+
  '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
 
  '''model''' - data transfer objects  
 
  '''model''' - data transfer objects  
 
  '''service''' (or rest) - web service entry points
 
  '''service''' (or rest) - web service entry points
Line 70: Line 71:
 
                     --> dao
 
                     --> dao
  
The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.
+
The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.
  
 
= High Level Design =
 
= High Level Design =
Line 76: Line 77:
 
==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==
 
==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==
  
IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons.  Managers are typically singletons.  This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.
+
IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons.  Managers are typically singletons.  This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.
  
 
===Spring Framework===
 
===Spring Framework===
Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library.  The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.
+
Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library.  The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.
  
 
===SDMX.IO IoC framework===
 
===SDMX.IO IoC framework===
In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class.  When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class.   
+
In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class.  When the class is not a singleton, instances can be registered with the '''FusionBeanStore''' class.   
  
 
A running application would request a singleton instance in the following way.
 
A running application would request a singleton instance in the following way.
Line 101: Line 102:
  
 
===Using Spring/SDMX.IO===
 
===Using Spring/SDMX.IO===
In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module.  
+
In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module.  
  
 
For example:
 
For example:
Line 115: Line 116:
 
  PublicationTableModule.register(); - classes to support Publication ables
 
  PublicationTableModule.register(); - classes to support Publication ables
 
    
 
    
The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore.  This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.
+
The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore.  This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.
  
 +
= Example Code =
 +
The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. 
  
= Example Code =
 
 
== Reading Structures ==
 
== Reading Structures ==
 +
* Register the SDMX-ML module as the response from the URL is in SDMX-ML format
 
* Creates an instance of the StructureReaderManager  (typically in an application this would be a singleton).  
 
* Creates an instance of the StructureReaderManager  (typically in an application this would be a singleton).  
* Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows
+
* Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows
 
* Convert the stream into SDMX structures
 
* Convert the stream into SDMX structures
 
* Loop the Dataflows and print the ID to the console
 
* Loop the Dataflows and print the ID to the console
  
 
  public static void main(String[] args) throws MalformedURLException {
 
  public static void main(String[] args) throws MalformedURLException {
 +
  SdmxMLModule.register();
 
   StructureReaderManager srm = new StructureReaderManagerImpl();
 
   StructureReaderManager srm = new StructureReaderManagerImpl();
 
   ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 
   ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
Line 131: Line 135:
 
   beans.getDataflows().forEach((flow) -> System.out.println(flow.getId()));
 
   beans.getDataflows().forEach((flow) -> System.out.println(flow.getId()));
 
  }
 
  }
 +
 +
== Writing / Transforming Structures ==
 +
The same code as  Reading Structures, but with the addition lines in BOLD
 +
 +
* Register the SDMX-JSON module as we are writing out in this format
 +
* Creates an instance of the StructureWriterManager (typically in an application this would be a singleton).
 +
* Writes the structures in FusionJson format to the system output
 +
 +
public static void main(String[] args) throws MalformedURLException {
 +
  SdmxMLModule.register();
 +
  '''SdmxJsonModule.register();'''
 +
  StructureReaderManager srm = new StructureReaderManagerImpl();
 +
  ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 +
  SdmxBeans beans = srm.parseStructures(rdl);
 +
  '''StructureWriterManager swm = new StructureWriterManagerImpl();'''
 +
  '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''
 +
}
 +
 +
 +
== Data Read ==
 +
Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. 
 +
 +
public static void main(String[] args) throws MalformedURLException {
 +
    //SDMX WEB SERVICE
 +
    String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
 +
    String dataflow = "dataflow/ECB/BSI/1.0";
 +
 +
    //Register modules for formats
 +
    SdmxMLModule.register();
 +
   
 +
    //Set up data reader and writer managers
 +
    DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
 +
    DataReaderManager drm = new DataReaderManagerImpl(dfm);
 +
   
 +
    //Get Structures from URL into an in-memory store
 +
    String structureURL = sdmxv2API + "/structure/"+dataflow+"?references=children";
 +
    SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());
 +
 +
    //Get Data
 +
    String dataQueryUrl = sdmxv2API+"/data/"+dataflow+"?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
 +
    ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
 +
    DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
 +
   
 +
    while(dre.moveNextDataset()) {
 +
      while(dre.moveNextKeyable()) {
 +
        System.out.println(dre.getCurrentKey().getShortCode());
 +
        while(dre.moveNextObservation()) {
 +
          Observation obs = dre.getCurrentObservation();
 +
          System.out.println(obs.getDimensionValue() + "=" + obs.getPrimaryMeasure());
 +
        }
 +
      }
 +
    }
 +
  }
 +
 +
== Data Transformation ==
 +
This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service.  This requires the registration of the REST classes.  In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts).  This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.
 +
 +
  public static void main(String[] args) throws MalformedURLException {
 +
    //SDMX WEB SERVICE
 +
    String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
 +
 +
    //Register modules for formats
 +
    SdmxMLModule.register();
 +
    SdmxJsonModule.register();
 +
   
 +
    //Manually register required instances
 +
    SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());
 +
    SingletonStore.registerInstance(new StructureQueryBuilderRest());
 +
    SingletonStore.registerInstance(new StructureReaderManagerImpl());
 +
   
 +
    //Set up data reader and writer managers
 +
    DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
 +
    DataReaderManager drm = new DataReaderManagerImpl(dfm);
 +
    DataWriterManager dwm = new DataWriterManagerImpl(dfm);
 +
   
 +
    //JSON output format requires access to the metadata so labels can be resolved
 +
    //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand
 +
    ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);
 +
    SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);
 +
 +
    //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is
 +
    //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object
 +
    SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);
 +
   
 +
    //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the
 +
    //structure retrieval interfaces - used by classes including the JSON writers
 +
    SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));
 +
   
 +
    //Get Data
 +
    String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
 +
    ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
 +
    DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
 +
   
 +
    //Create Writer
 +
    ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);
 +
   
 +
    //Transform Data
 +
    DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true);
 +
    DataTransformationUtil.copyData(dre, dwe, transformOptions);
 +
  }

Latest revision as of 09:40, 24 July 2023

Architecture

Modules

The sdmx-io modules are as follows

Interface Modules

These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.

fusion-api  - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception
fusion-api-dao - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects
fusion-api-sdmx - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules
fusion-api-service - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies

Implementation Modules

These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules

fusion-sdmx-im - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)
fusion-core-data - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores
fusion-core-sdmx - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features 
fusion-core-structure - contains core code for processing structural metadata 
fusion-core-xlsx - contains code to read and write structures and data in Excel 

Format Modules

These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data

fusion-sdmx-csv - reading and writing SDMX-CSV format
fusion-sdmx-edi - reading and writing SDMX-EDI format
fusion-sdmx-json - reading and writing SDMX-JSON format
fusion-sdmx-ml reading and writing SDMX-XML

Feature Modules

These modules provide feature specific code

fusion-core-dao - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option
fusion-audit - audit events and centralised error handling
fusion-sdmx-kafka - connecting to and sending/receiving messages to Apache Kafka
fusion-sdmx-publication - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition
fusion-sdmx-reporttemplate - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks
fusion-sdmx-rest - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service
fusion-sdmx-search - Lucene search engine for structures
fusion-sdmx-security - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)
fusion-xl - exposes web services required by the FusionXL Excel Plugin

Utility Modules

Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)

fusion-utils - generic utility classes, not specific to SDMX
fusion-utils-hibernate - implementation code to use Hibernate as a means to connect to the DAO
fusion-utils-http - calling HTTP web services
fusion-utils-json - JSON related utility classes
fusion-utils-mail - email utility classes
fusion-utils-mongodb  - implementation code to use MongoDB a means to store Objects
fusion-utils-sdmx - SDMX Specific utility classes
fusion-utils-spring - spring related utility classes
fusion-utils-xlsx - reading/writing Excel workbooks related utility classes

Java Packages

Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function.

The generic packages are as follows.

api - contains interfaces relevant to the module 
application - contains code specific to the building of an application - this will contain any IFusionModule specific to this module (described later)
constant - contains Java enums and java classes that contain public static constants
engine - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine 
factory - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
filter  - REST web service filters (defined in web.xml) - filters are called before the web service
manager - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
model - data transfer objects 
service (or rest) - web service entry points
util - utility classes, typically containing public static methods

The general logical flow of a system is the following

service --> manager --> engine
                    --> dao

The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.

High Level Design

Inversion of Control (IoC)

IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.

Spring Framework

Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.

SDMX.IO IoC framework

In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the SingletonStore class. When the class is not a singleton, instances can be registered with the FusionBeanStore class.

A running application would request a singleton instance in the following way.

private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);

This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.

There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.

SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -> {
  infoRetreival = bean;
});

If the application wants multiple instances of an interface, then the FusionBeanStore is used

Collection<StructureWriterFactory> allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);

Using Spring/SDMX.IO

In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement IFusionModule, these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module.

For example:

CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager
SdmxMLModule.register();  - register SDMX-XML data and structure format readers and writers
SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers
FusionXLModule.register(); - register FusionXL classes for the excel plugin
SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers
FusionJsonModule.register();  - register Fusion-JSON data and structure format readers and writers
SdmxCsvModule.register();  - register SDMX-CSV data format readers and writers
FusionCoreDataModule.register();  - register SDMX-EDI data and structure format readers and writers
XLSXModule.register();  - register reader and writer for Excel
PublicationTableModule.register(); - classes to support Publication ables
 

The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class SpringBeansContainer which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.

Example Code

The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation.

Reading Structures

  • Register the SDMX-ML module as the response from the URL is in SDMX-ML format
  • Creates an instance of the StructureReaderManager (typically in an application this would be a singleton).
  • Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows
  • Convert the stream into SDMX structures
  • Loop the Dataflows and print the ID to the console
public static void main(String[] args) throws MalformedURLException {
 SdmxMLModule.register();
 StructureReaderManager srm = new StructureReaderManagerImpl();
 ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 SdmxBeans beans = srm.parseStructures(rdl);
 beans.getDataflows().forEach((flow) -> System.out.println(flow.getId()));
}

Writing / Transforming Structures

The same code as Reading Structures, but with the addition lines in BOLD

  • Register the SDMX-JSON module as we are writing out in this format
  • Creates an instance of the StructureWriterManager (typically in an application this would be a singleton).
  • Writes the structures in FusionJson format to the system output
public static void main(String[] args) throws MalformedURLException {
 SdmxMLModule.register();
 SdmxJsonModule.register();
 StructureReaderManager srm = new StructureReaderManagerImpl();
 ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 SdmxBeans beans = srm.parseStructures(rdl);
 StructureWriterManager swm = new StructureWriterManagerImpl();
 swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);
}


Data Read

Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts.

public static void main(String[] args) throws MalformedURLException {
   //SDMX WEB SERVICE
   String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
   String dataflow = "dataflow/ECB/BSI/1.0";

   //Register modules for formats
   SdmxMLModule.register();
   	
   //Set up data reader and writer managers
   DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
   DataReaderManager drm = new DataReaderManagerImpl(dfm);
   	
   //Get Structures from URL into an in-memory store
   String structureURL = sdmxv2API + "/structure/"+dataflow+"?references=children";
   SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());

   //Get Data
   String dataQueryUrl = sdmxv2API+"/data/"+dataflow+"?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
   ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
   DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
   	
   while(dre.moveNextDataset()) {
     while(dre.moveNextKeyable()) {
       System.out.println(dre.getCurrentKey().getShortCode());
       while(dre.moveNextObservation()) {
         Observation obs = dre.getCurrentObservation();
         System.out.println(obs.getDimensionValue() + "=" + obs.getPrimaryMeasure());
       }
     }	
   }
 }

Data Transformation

This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.

 public static void main(String[] args) throws MalformedURLException {
   //SDMX WEB SERVICE
   String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";

   //Register modules for formats
   SdmxMLModule.register();
   SdmxJsonModule.register();
   	
   //Manually register required instances		
   SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());
   SingletonStore.registerInstance(new StructureQueryBuilderRest());
   SingletonStore.registerInstance(new StructureReaderManagerImpl());
   	
   //Set up data reader and writer managers
   DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
   DataReaderManager drm = new DataReaderManagerImpl(dfm);
   DataWriterManager dwm = new DataWriterManagerImpl(dfm);
   	
   //JSON output format requires access to the metadata so labels can be resolved
   //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand
   ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);
   SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);

   //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is 
   //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object
   SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);
   	 
   //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the 
   //structure retrieval interfaces - used by classes including the JSON writers
   SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));
   	
   //Get Data
   String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
   ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
   DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
   	
   //Create Writer
   ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);
   	
   //Transform Data
   DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); 
   DataTransformationUtil.copyData(dre, dwe, transformOptions);
 }