Difference between revisions of "DRAFT SDMX-CORE"
(→Interface Modules) |
(→Using Spring/SDMX.IO) |
||
(18 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
− | = | + | = Architecture = |
== Modules == | == Modules == | ||
The sdmx-io modules are as follows | The sdmx-io modules are as follows | ||
Line 6: | Line 6: | ||
'''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception | '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception | ||
− | '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on | + | '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects |
− | '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross | + | '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules |
− | '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service | + | '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies |
===Implementation Modules=== | ===Implementation Modules=== | ||
Line 19: | Line 19: | ||
===Format Modules=== | ===Format Modules=== | ||
− | These modules contains Java classes for reading and writing SDMX | + | These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data |
'''fusion-sdmx-csv''' - reading and writing SDMX-CSV format | '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format | ||
Line 30: | Line 30: | ||
'''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option | '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option | ||
'''fusion-audit''' - audit events and centralised error handling | '''fusion-audit''' - audit events and centralised error handling | ||
− | '''fusion-sdmx-kafka''' - connecting to and sending/ | + | '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka |
− | '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and | + | '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition |
− | '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and | + | '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks |
'''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service | '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service | ||
'''fusion-sdmx-search''' - Lucene search engine for structures | '''fusion-sdmx-search''' - Lucene search engine for structures | ||
'''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl) | '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl) | ||
− | '''fusion-xl''' - exposes web services required by the FusionXL Excel | + | '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin |
===Utility Modules=== | ===Utility Modules=== | ||
Line 44: | Line 44: | ||
'''fusion-utils-http''' - calling HTTP web services | '''fusion-utils-http''' - calling HTTP web services | ||
'''fusion-utils-json''' - JSON related utility classes | '''fusion-utils-json''' - JSON related utility classes | ||
− | '''fusion-utils-mail''' - email | + | '''fusion-utils-mail''' - email utility classes |
'''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects | '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects | ||
− | '''fusion-utils-sdmx''' - SDMX Specific | + | '''fusion-utils-sdmx''' - SDMX Specific utility classes |
'''fusion-utils-spring''' - spring related utility classes | '''fusion-utils-spring''' - spring related utility classes | ||
'''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes | '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes | ||
== Java Packages == | == Java Packages == | ||
− | Each module is partitioned into Java packages, with the exception of the utility modules, which typically | + | Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. |
− | The generic packages | + | The generic packages are as follows. |
− | '''api''' - contains interfaces | + | '''api''' - contains interfaces relevant to the module |
+ | '''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later) | ||
'''constant''' - contains Java enums and java classes that contain public static constants | '''constant''' - contains Java enums and java classes that contain public static constants | ||
'''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine | '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine | ||
− | '''factory''' - singleton classes that are | + | '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework |
'''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service | '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service | ||
− | '''manager''' - the main entry point for all high level functions, the manager's job is to | + | '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer |
'''model''' - data transfer objects | '''model''' - data transfer objects | ||
'''service''' (or rest) - web service entry points | '''service''' (or rest) - web service entry points | ||
'''util''' - utility classes, typically containing public static methods | '''util''' - utility classes, typically containing public static methods | ||
+ | |||
+ | The general logical flow of a system is the following | ||
+ | |||
+ | service --> manager --> engine | ||
+ | --> dao | ||
+ | |||
+ | The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos. | ||
+ | |||
+ | = High Level Design = | ||
+ | |||
+ | ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)== | ||
+ | |||
+ | IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework. | ||
+ | |||
+ | ===Spring Framework=== | ||
+ | Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions. | ||
+ | |||
+ | ===SDMX.IO IoC framework=== | ||
+ | In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton, instances can be registered with the '''FusionBeanStore''' class. | ||
+ | |||
+ | A running application would request a singleton instance in the following way. | ||
+ | |||
+ | private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class); | ||
+ | |||
+ | This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched. | ||
+ | |||
+ | There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered. | ||
+ | |||
+ | SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -> { | ||
+ | infoRetreival = bean; | ||
+ | }); | ||
+ | |||
+ | If the application wants multiple instances of an interface, then the FusionBeanStore is used | ||
+ | |||
+ | Collection<StructureWriterFactory> allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class); | ||
+ | |||
+ | ===Using Spring/SDMX.IO=== | ||
+ | In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. | ||
+ | |||
+ | For example: | ||
+ | CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager | ||
+ | SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers | ||
+ | SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers | ||
+ | FusionXLModule.register(); - register FusionXL classes for the excel plugin | ||
+ | SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers | ||
+ | FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers | ||
+ | SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers | ||
+ | FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers | ||
+ | XLSXModule.register(); - register reader and writer for Excel | ||
+ | PublicationTableModule.register(); - classes to support Publication ables | ||
+ | |||
+ | The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes. | ||
+ | |||
+ | = Example Code = | ||
+ | The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. | ||
+ | |||
+ | == Reading Structures == | ||
+ | * Register the SDMX-ML module as the response from the URL is in SDMX-ML format | ||
+ | * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). | ||
+ | * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows | ||
+ | * Convert the stream into SDMX structures | ||
+ | * Loop the Dataflows and print the ID to the console | ||
+ | |||
+ | public static void main(String[] args) throws MalformedURLException { | ||
+ | SdmxMLModule.register(); | ||
+ | StructureReaderManager srm = new StructureReaderManagerImpl(); | ||
+ | ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow")); | ||
+ | SdmxBeans beans = srm.parseStructures(rdl); | ||
+ | beans.getDataflows().forEach((flow) -> System.out.println(flow.getId())); | ||
+ | } | ||
+ | |||
+ | == Writing / Transforming Structures == | ||
+ | The same code as Reading Structures, but with the addition lines in BOLD | ||
+ | |||
+ | * Register the SDMX-JSON module as we are writing out in this format | ||
+ | * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). | ||
+ | * Writes the structures in FusionJson format to the system output | ||
+ | |||
+ | public static void main(String[] args) throws MalformedURLException { | ||
+ | SdmxMLModule.register(); | ||
+ | '''SdmxJsonModule.register();''' | ||
+ | StructureReaderManager srm = new StructureReaderManagerImpl(); | ||
+ | ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow")); | ||
+ | SdmxBeans beans = srm.parseStructures(rdl); | ||
+ | '''StructureWriterManager swm = new StructureWriterManagerImpl();''' | ||
+ | '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);''' | ||
+ | } | ||
+ | |||
+ | |||
+ | == Data Read == | ||
+ | Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. | ||
+ | |||
+ | public static void main(String[] args) throws MalformedURLException { | ||
+ | //SDMX WEB SERVICE | ||
+ | String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2"; | ||
+ | String dataflow = "dataflow/ECB/BSI/1.0"; | ||
+ | |||
+ | //Register modules for formats | ||
+ | SdmxMLModule.register(); | ||
+ | |||
+ | //Set up data reader and writer managers | ||
+ | DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance()); | ||
+ | DataReaderManager drm = new DataReaderManagerImpl(dfm); | ||
+ | |||
+ | //Get Structures from URL into an in-memory store | ||
+ | String structureURL = sdmxv2API + "/structure/"+dataflow+"?references=children"; | ||
+ | SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl()); | ||
+ | |||
+ | //Get Data | ||
+ | String dataQueryUrl = sdmxv2API+"/data/"+dataflow+"?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES"; | ||
+ | ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl); | ||
+ | DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null); | ||
+ | |||
+ | while(dre.moveNextDataset()) { | ||
+ | while(dre.moveNextKeyable()) { | ||
+ | System.out.println(dre.getCurrentKey().getShortCode()); | ||
+ | while(dre.moveNextObservation()) { | ||
+ | Observation obs = dre.getCurrentObservation(); | ||
+ | System.out.println(obs.getDimensionValue() + "=" + obs.getPrimaryMeasure()); | ||
+ | } | ||
+ | } | ||
+ | } | ||
+ | } | ||
+ | |||
+ | == Data Transformation == | ||
+ | This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information. | ||
+ | |||
+ | public static void main(String[] args) throws MalformedURLException { | ||
+ | //SDMX WEB SERVICE | ||
+ | String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2"; | ||
+ | |||
+ | //Register modules for formats | ||
+ | SdmxMLModule.register(); | ||
+ | SdmxJsonModule.register(); | ||
+ | |||
+ | //Manually register required instances | ||
+ | SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl()); | ||
+ | SingletonStore.registerInstance(new StructureQueryBuilderRest()); | ||
+ | SingletonStore.registerInstance(new StructureReaderManagerImpl()); | ||
+ | |||
+ | //Set up data reader and writer managers | ||
+ | DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance()); | ||
+ | DataReaderManager drm = new DataReaderManagerImpl(dfm); | ||
+ | DataWriterManager dwm = new DataWriterManagerImpl(dfm); | ||
+ | |||
+ | //JSON output format requires access to the metadata so labels can be resolved | ||
+ | //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand | ||
+ | ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0); | ||
+ | SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager); | ||
+ | |||
+ | //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is | ||
+ | //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object | ||
+ | SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval); | ||
+ | |||
+ | //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the | ||
+ | //structure retrieval interfaces - used by classes including the JSON writers | ||
+ | SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager)); | ||
+ | |||
+ | //Get Data | ||
+ | String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES"; | ||
+ | ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl); | ||
+ | DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null); | ||
+ | |||
+ | //Create Writer | ||
+ | ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out); | ||
+ | |||
+ | //Transform Data | ||
+ | DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); | ||
+ | DataTransformationUtil.copyData(dre, dwe, transformOptions); | ||
+ | } |
Latest revision as of 09:40, 24 July 2023
Architecture
Modules
The sdmx-io modules are as follows
Interface Modules
These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.
fusion-api - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception fusion-api-dao - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects fusion-api-sdmx - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules fusion-api-service - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies
Implementation Modules
These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules
fusion-sdmx-im - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc) fusion-core-data - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores fusion-core-sdmx - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features fusion-core-structure - contains core code for processing structural metadata fusion-core-xlsx - contains code to read and write structures and data in Excel
Format Modules
These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data
fusion-sdmx-csv - reading and writing SDMX-CSV format fusion-sdmx-edi - reading and writing SDMX-EDI format fusion-sdmx-json - reading and writing SDMX-JSON format fusion-sdmx-ml reading and writing SDMX-XML
Feature Modules
These modules provide feature specific code
fusion-core-dao - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option fusion-audit - audit events and centralised error handling fusion-sdmx-kafka - connecting to and sending/receiving messages to Apache Kafka fusion-sdmx-publication - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition fusion-sdmx-reporttemplate - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks fusion-sdmx-rest - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service fusion-sdmx-search - Lucene search engine for structures fusion-sdmx-security - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl) fusion-xl - exposes web services required by the FusionXL Excel Plugin
Utility Modules
Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)
fusion-utils - generic utility classes, not specific to SDMX fusion-utils-hibernate - implementation code to use Hibernate as a means to connect to the DAO fusion-utils-http - calling HTTP web services fusion-utils-json - JSON related utility classes fusion-utils-mail - email utility classes fusion-utils-mongodb - implementation code to use MongoDB a means to store Objects fusion-utils-sdmx - SDMX Specific utility classes fusion-utils-spring - spring related utility classes fusion-utils-xlsx - reading/writing Excel workbooks related utility classes
Java Packages
Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function.
The generic packages are as follows.
api - contains interfaces relevant to the module application - contains code specific to the building of an application - this will contain any IFusionModule specific to this module (described later) constant - contains Java enums and java classes that contain public static constants engine - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine factory - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework filter - REST web service filters (defined in web.xml) - filters are called before the web service manager - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer model - data transfer objects service (or rest) - web service entry points util - utility classes, typically containing public static methods
The general logical flow of a system is the following
service --> manager --> engine --> dao
The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.
High Level Design
Inversion of Control (IoC)
IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.
Spring Framework
Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.
SDMX.IO IoC framework
In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the SingletonStore class. When the class is not a singleton, instances can be registered with the FusionBeanStore class.
A running application would request a singleton instance in the following way.
private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);
This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.
There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.
SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -> { infoRetreival = bean; });
If the application wants multiple instances of an interface, then the FusionBeanStore is used
Collection<StructureWriterFactory> allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);
Using Spring/SDMX.IO
In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement IFusionModule, these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module.
For example:
CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers FusionXLModule.register(); - register FusionXL classes for the excel plugin SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers XLSXModule.register(); - register reader and writer for Excel PublicationTableModule.register(); - classes to support Publication ables
The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class SpringBeansContainer which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.
Example Code
The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation.
Reading Structures
- Register the SDMX-ML module as the response from the URL is in SDMX-ML format
- Creates an instance of the StructureReaderManager (typically in an application this would be a singleton).
- Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows
- Convert the stream into SDMX structures
- Loop the Dataflows and print the ID to the console
public static void main(String[] args) throws MalformedURLException { SdmxMLModule.register(); StructureReaderManager srm = new StructureReaderManagerImpl(); ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow")); SdmxBeans beans = srm.parseStructures(rdl); beans.getDataflows().forEach((flow) -> System.out.println(flow.getId())); }
Writing / Transforming Structures
The same code as Reading Structures, but with the addition lines in BOLD
- Register the SDMX-JSON module as we are writing out in this format
- Creates an instance of the StructureWriterManager (typically in an application this would be a singleton).
- Writes the structures in FusionJson format to the system output
public static void main(String[] args) throws MalformedURLException { SdmxMLModule.register(); SdmxJsonModule.register(); StructureReaderManager srm = new StructureReaderManagerImpl(); ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL("https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow")); SdmxBeans beans = srm.parseStructures(rdl); StructureWriterManager swm = new StructureWriterManagerImpl(); swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out); }
Data Read
Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts.
public static void main(String[] args) throws MalformedURLException { //SDMX WEB SERVICE String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2"; String dataflow = "dataflow/ECB/BSI/1.0"; //Register modules for formats SdmxMLModule.register(); //Set up data reader and writer managers DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance()); DataReaderManager drm = new DataReaderManagerImpl(dfm); //Get Structures from URL into an in-memory store String structureURL = sdmxv2API + "/structure/"+dataflow+"?references=children"; SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl()); //Get Data String dataQueryUrl = sdmxv2API+"/data/"+dataflow+"?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES"; ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl); DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null); while(dre.moveNextDataset()) { while(dre.moveNextKeyable()) { System.out.println(dre.getCurrentKey().getShortCode()); while(dre.moveNextObservation()) { Observation obs = dre.getCurrentObservation(); System.out.println(obs.getDimensionValue() + "=" + obs.getPrimaryMeasure()); } } } }
Data Transformation
This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.
public static void main(String[] args) throws MalformedURLException { //SDMX WEB SERVICE String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2"; //Register modules for formats SdmxMLModule.register(); SdmxJsonModule.register(); //Manually register required instances SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl()); SingletonStore.registerInstance(new StructureQueryBuilderRest()); SingletonStore.registerInstance(new StructureReaderManagerImpl()); //Set up data reader and writer managers DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance()); DataReaderManager drm = new DataReaderManagerImpl(dfm); DataWriterManager dwm = new DataWriterManagerImpl(dfm); //JSON output format requires access to the metadata so labels can be resolved //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0); SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager); //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval); //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the //structure retrieval interfaces - used by classes including the JSON writers SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager)); //Get Data String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES"; ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl); DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null); //Create Writer ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out); //Transform Data DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); DataTransformationUtil.copyData(dre, dwe, transformOptions); }