DRAFT SDMX-CORE - Revision history

Mnelson: /* Using Spring/SDMX.IO */

2023-07-24T16:40:33Z

‎Using Spring/SDMX.IO

Mnelson: /* SDMX.IO IoC framework */

2023-07-24T16:39:15Z

‎SDMX.IO IoC framework

Mnelson: /* Java Packages */

2023-07-24T16:38:24Z

‎Java Packages

Mnelson: /* Java Packages */

2023-07-24T16:37:36Z

‎Java Packages

Mnelson at 16:33, 24 July 2023

2023-07-24T16:33:55Z

Mnelson at 16:29, 24 July 2023

2023-07-24T16:29:22Z

Mnelson: /* Example Code */

2023-07-24T16:27:58Z

‎Example Code

Mnelson: /* Data Transformation */

2023-07-24T16:24:59Z

‎Data Transformation

Mnelson: /* Data Transformation */

2023-07-24T16:24:24Z

‎Data Transformation

Mnelson: /* Data Transformation */

2023-07-24T16:24:08Z

‎Data Transformation

@@ Line 116: / Line 116: @@
   PublicationTableModule.register(); - classes to support Publication ables
-The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore.  This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.
+The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore.  This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.
 = Example Code =

@@ Line 83: / Line 83: @@
 ===SDMX.IO IoC framework===
-In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class.  When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class.
+In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class.  When the class is not a singleton, instances can be registered with the '''FusionBeanStore''' class.
 A running application would request a singleton instance in the following way.

@@ Line 71: / Line 71: @@
                       --> dao
-The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by making calls to engines, daos.
+The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.
 = High Level Design =

@@ Line 56: / Line 56: @@
   '''api''' - contains interfaces relevant to the module
   '''constant''' - contains Java enums and java classes that contain public static constants
   '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine

@@ Line 6: / Line 6: @@
   '''fusion-api'''  - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception
-  '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects
+  '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects
-  '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules
+  '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules
-  '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies
+  '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies
 ===Implementation Modules===
@@ Line 19: / Line 19: @@
 ===Format Modules===
-These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data
+These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data
   '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format
@@ Line 30: / Line 30: @@
   '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option
   '''fusion-audit''' - audit events and centralised error handling
-  '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka
+  '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka
-  '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition
+  '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition
-  '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks
+  '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks
   '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service
   '''fusion-sdmx-search''' - Lucene search engine for structures
   '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)
-  '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin
+  '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin
 ===Utility Modules===
@@ Line 44: / Line 44: @@
   '''fusion-utils-http''' - calling HTTP web services
   '''fusion-utils-json''' - JSON related utility classes
-  '''fusion-utils-mail''' - email utiliity classes
+  '''fusion-utils-mail''' - email utility classes
   '''fusion-utils-mongodb'''  - implementation code to use MongoDB a means to store Objects
-  '''fusion-utils-sdmx''' - SDMX Specific utilty classes
+  '''fusion-utils-sdmx''' - SDMX Specific utility classes
   '''fusion-utils-spring''' - spring related utility classes
   '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes
 == Java Packages ==
-Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function.
+Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function.
 The generic packages are as follows.
-  '''api''' - contains interfaces relevent to the module
+  '''api''' - contains interfaces relevant to the module
   '''constant''' - contains Java enums and java classes that contain public static constants
   '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine
-  '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
+  '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework
   '''filter'''  - REST web service filters (defined in web.xml) - filters are called before the web service
-  '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
+  '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer
   '''model''' - data transfer objects
   '''service''' (or rest) - web service entry points
@@ Line 70: / Line 70: @@
                       --> dao
-The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.
+The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface.  The manager will ensure the arguments are valid and the user has permission to perform the action.  The manager then provides the workflow required to fulfil the task, by making calls to engines, daos.
 = High Level Design =
@@ Line 76: / Line 76: @@
 ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==
-IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons.  Managers are typically singletons.  This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.
+IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons.  Managers are typically singletons.  This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.
 ===Spring Framework===
-Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library.  The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.
+Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library.  The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.
 ===SDMX.IO IoC framework===
@@ Line 101: / Line 101: @@
 ===Using Spring/SDMX.IO===
-In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module.
+In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module.
 For example:
@@ Line 124: / Line 124: @@
 * Register the SDMX-ML module as the response from the URL is in SDMX-ML format
 * Creates an instance of the StructureReaderManager  (typically in an application this would be a singleton).
-* Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows
+* Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows
 * Convert the stream into SDMX structures
 * Loop the Dataflows and print the ID to the console
@@ Line 139: / Line 139: @@
 The same code as  Reading Structures, but with the addition lines in BOLD
-* Register the SDMX-JSON module as we are wriring out in this format
+* Register the SDMX-JSON module as we are writing out in this format
 * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton).
 * Writes the structures in FusionJson format to the system output
@@ Line 190: / Line 190: @@
 == Data Transformation ==
-This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service.  This requires the registration of the REST
+This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service.  This requires the registration of the REST classes.  In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts).  This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.
    public static void main(String[] args) throws MalformedURLException {

@@ Line 1: / Line 1: @@
-= Architectire =
+= Architecture =
 == Modules ==
 The sdmx-io modules are as follows

@@ Line 192: / Line 192: @@
 the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.
-    public static void main(String[] args) throws MalformedURLException {
+  public static void main(String[] args) throws MalformedURLException {
-    	//SDMX WEB SERVICE
+    //SDMX WEB SERVICE
-    	String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
+    String sdmxv2API = "https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2";
-    	//Register modules for formats
+    //Register modules for formats
-    	SdmxMLModule.register();
+    SdmxMLModule.register();
-    	SdmxJsonModule.register();
+    SdmxJsonModule.register();
-    	//Manually register required instances
+    //Manually register required instances
-    	SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());
+    SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());
-    	SingletonStore.registerInstance(new StructureQueryBuilderRest());
+    SingletonStore.registerInstance(new StructureQueryBuilderRest());
-    	SingletonStore.registerInstance(new StructureReaderManagerImpl());
+    SingletonStore.registerInstance(new StructureReaderManagerImpl());
-    	//Set up data reader and writer managers
+    //Set up data reader and writer managers
-    	DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
+    DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());
-    	DataReaderManager drm = new DataReaderManagerImpl(dfm);
+    DataReaderManager drm = new DataReaderManagerImpl(dfm);
-    	DataWriterManager dwm = new DataWriterManagerImpl(dfm);
+    DataWriterManager dwm = new DataWriterManagerImpl(dfm);
-    	//JSON output format requires access to the metadata so labels can be resolved
+    //JSON output format requires access to the metadata so labels can be resolved
-    	//Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand
+    //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand
-    	ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);
+    ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);
-    	SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);
+    SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);
-    	//Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is
+    //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is
-    	//required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object
+    //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object
-    	SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);
+    SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);
-    	//Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the
+    //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the
-    	//structure retrieval interfaces - used by classes including the JSON writers
+    //structure retrieval interfaces - used by classes including the JSON writers
-    	SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));
+    SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));
-    	//Get Data
+    //Get Data
-    	String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
+    String dataQueryUrl = sdmxv2API+"/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&c%5BREF_AREA%5D=DE,ES";
-    	ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
+    ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);
-    	DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
+    DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);
-    	//Create Writer
+    //Create Writer
-    	ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);
+    ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);
-    	//Transform Data
+    //Transform Data
-    	DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true);
+    DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true);
-    	DataTransformationUtil.copyData(dre, dwe, transformOptions);
+    DataTransformationUtil.copyData(dre, dwe, transformOptions);
+   }

← Older revision		Revision as of 16:24, 24 July 2023
Line 234:		Line 234:
	DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true);		DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true);
	DataTransformationUtil.copyData(dre, dwe, transformOptions);		DataTransformationUtil.copyData(dre, dwe, transformOptions);
−	}	+	}