DRAFT SDMX-CORE

Architecture

Modules

The sdmx-io modules are as follows

Interface Modules

These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.

fusion-api  - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception
fusion-api-dao - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects
fusion-api-sdmx - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules
fusion-api-service - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies

Implementation Modules

These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules

fusion-sdmx-im - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)
fusion-core-data - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores
fusion-core-sdmx - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features 
fusion-core-structure - contains core code for processing structural metadata 
fusion-core-xlsx - contains code to read and write structures and data in Excel 

Format Modules

These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data

fusion-sdmx-csv - reading and writing SDMX-CSV format
fusion-sdmx-edi - reading and writing SDMX-EDI format
fusion-sdmx-json - reading and writing SDMX-JSON format
fusion-sdmx-ml reading and writing SDMX-XML

Feature Modules

These modules provide feature specific code

fusion-core-dao - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option
fusion-audit - audit events and centralised error handling
fusion-sdmx-kafka - connecting to and sending/receiving messages to Apache Kafka
fusion-sdmx-publication - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition
fusion-sdmx-reporttemplate - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks
fusion-sdmx-rest - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service
fusion-sdmx-search - Lucene search engine for structures
fusion-sdmx-security - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)
fusion-xl - exposes web services required by the FusionXL Excel Plugin

Utility Modules

Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)

fusion-utils - generic utility classes, not specific to SDMX
fusion-utils-hibernate - implementation code to use Hibernate as a means to connect to the DAO
fusion-utils-http - calling HTTP web services
fusion-utils-json - JSON related utility classes
fusion-utils-mail - email utility classes
fusion-utils-mongodb  - implementation code to use MongoDB a means to store Objects
fusion-utils-sdmx - SDMX Specific utility classes
fusion-utils-spring - spring related utility classes
fusion-utils-xlsx - reading/writing Excel workbooks related utility classes

Java Packages

Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function.

The generic packages are as follows.

api - contains interfaces relevant to the module 
application - contains code specific to the building of an application - this will contain any IFusionModule specific to this module (described later)
constant - contains Java enums and java classes that contain public static constants
engine - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine 
factory - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
filter  - REST web service filters (defined in web.xml) - filters are called before the web service
manager - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
model - data transfer objects 
service (or rest) - web service entry points
util - utility classes, typically containing public static methods

The general logical flow of a system is the following

service --> manager --> engine
                    --> dao

The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.

High Level Design

Inversion of Control (IoC)

IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.

Spring Framework

Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.

SDMX.IO IoC framework

In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the SingletonStore class. When the class is not a singleton, instances can be registered with the FusionBeanStore class.

A running application would request a singleton instance in the following way.

private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);

This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.

There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.

SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -> {
  infoRetreival = bean;
});

If the application wants multiple instances of an interface, then the FusionBeanStore is used

Collection<StructureWriterFactory> allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);

Using Spring/SDMX.IO

In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement IFusionModule, these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module.

For example:

CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager
SdmxMLModule.register();  - register SDMX-XML data and structure format readers and writers
SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers
FusionXLModule.register(); - register FusionXL classes for the excel plugin
SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers
FusionJsonModule.register();  - register Fusion-JSON data and structure format readers and writers
SdmxCsvModule.register();  - register SDMX-CSV data format readers and writers
FusionCoreDataModule.register();  - register SDMX-EDI data and structure format readers and writers
XLSXModule.register();  - register reader and writer for Excel
PublicationTableModule.register(); - classes to support Publication ables
 

The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class SpringBeansContainer which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.

Example Code

The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation.

Reading Structures

• Register the SDMX-ML module as the response from the URL is in SDMX-ML format
• Creates an instance of the StructureReaderManager (typically in an application this would be a singleton).
• Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows
• Convert the stream into SDMX structures
• Loop the Dataflows and print the ID to the console

public static void main(String[] args) throws MalformedURLException {
 SdmxMLModule.register();
 StructureReaderManager srm = new StructureReaderManagerImpl();
 ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 SdmxBeans beans = srm.parseStructures(rdl);
 beans.getDataflows().forEach((flow) -> System.out.println(flow.getId()));
}

Writing / Transforming Structures

The same code as Reading Structures, but with the addition lines in BOLD

• Register the SDMX-JSON module as we are writing out in this format
• Creates an instance of the StructureWriterManager (typically in an application this would be a singleton).
• Writes the structures in FusionJson format to the system output

public static void main(String[] args) throws MalformedURLException {
 SdmxMLModule.register();
 SdmxJsonModule.register();
 StructureReaderManager srm = new StructureReaderManagerImpl();
 ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow"));
 SdmxBeans beans = srm.parseStructures(rdl);
 StructureWriterManager swm = new StructureWriterManagerImpl();
 swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);
}

Data Read

Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts.

public static void main(String[] args) throws MalformedURLException {
   //SDMX WEB SERVICE
   String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
   String dataflow = "dataflow/ECB/BSI/1.0";

   //Register modules for formats
   SdmxMLModule.register();
   	
   //Set up data reader and writer managers
   DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
   DataReaderManager drm = new DataReaderManagerImpl(dfm);
   	
   //Get Structures from URL into an in-memory store
   String structureURL = sdmxv2API + "/structure/"+dataflow+"?references=children";
   SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());

   //Get Data
   String dataQueryUrl = sdmxv2API+"/data/"+dataflow+"?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
   ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
   DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
   	
   while(dre.moveNextDataset()) {
     while(dre.moveNextKeyable()) {
       System.out.println(dre.getCurrentKey().getShortCode());
       while(dre.moveNextObservation()) {
         Observation obs = dre.getCurrentObservation();
         System.out.println(obs.getDimensionValue() + "=" + obs.getPrimaryMeasure());
       }
     }	
   }
 }

Data Transformation

This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.

 public static void main(String[] args) throws MalformedURLException {
   //SDMX WEB SERVICE
   String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";

   //Register modules for formats
   SdmxMLModule.register();
   SdmxJsonModule.register();
   	
   //Manually register required instances		
   SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());
   SingletonStore.registerInstance(new StructureQueryBuilderRest());
   SingletonStore.registerInstance(new StructureReaderManagerImpl());
   	
   //Set up data reader and writer managers
   DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
   DataReaderManager drm = new DataReaderManagerImpl(dfm);
   DataWriterManager dwm = new DataWriterManagerImpl(dfm);
   	
   //JSON output format requires access to the metadata so labels can be resolved
   //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand
   ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);
   SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);

   //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is 
   //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object
   SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);
   	 
   //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the 
   //structure retrieval interfaces - used by classes including the JSON writers
   SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));
   	
   //Get Data
   String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
   ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
   DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
   	
   //Create Writer
   ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);
   	
   //Transform Data
   DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); 
   DataTransformationUtil.copyData(dre, dwe, transformOptions);
 }