https://fmrwiki.sdmxcloud.org/api.php?action=feedcontributions&feedformat=atom&user=Mnelson 2026-05-27T02:32:04Z User contributions MediaWiki 1.32.0 https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7095 2023-07-24T16:40:33Z

<p>Mnelson: /* Using Spring/SDMX.IO */</p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utility classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utility classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevant to the module <br /> '''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later)<br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton, instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans container, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are writing out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7094 2023-07-24T16:39:15Z

<p>Mnelson: /* SDMX.IO IoC framework */</p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utility classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utility classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevant to the module <br /> '''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later)<br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton, instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are writing out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7093 2023-07-24T16:38:24Z

<p>Mnelson: /* Java Packages */</p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utility classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utility classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevant to the module <br /> '''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later)<br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines and/or daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are writing out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7092 2023-07-24T16:37:36Z

<p>Mnelson: /* Java Packages */</p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utility classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utility classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevant to the module <br /> '''application''' - contains code specific to the building of an application - this will contain any '''IFusionModule''' specific to this module (described later)<br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are writing out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7091 2023-07-24T16:33:55Z

<p>Mnelson: </p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross multiple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service dependencies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structural Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/receiving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and business logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and business logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Plugin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utility classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utility classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically separate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevant to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responsible for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to verify the request (i.e. a security check) and manage the process of fulfilling the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with minimal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an appropriate model object (or objects) which contain a consolidated collection of arguments, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by making calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged together either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependencies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevant to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing multiple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are writing out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7090 2023-07-24T16:29:22Z

<p>Mnelson: </p> <hr /> <div>= Architecture =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST<br /> classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires <br /> the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7089 2023-07-24T16:27:58Z

<p>Mnelson: /* Example Code */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> The following example code blocks are all in one Java main method - which in a real system would not be the case, most of the code in these examples is setting up the system (registering modules and singletons), which is a one time system set up operation. <br /> <br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST<br /> classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires <br /> the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7088 2023-07-24T16:24:59Z

<p>Mnelson: /* Data Transformation */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST<br /> classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires <br /> the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7087 2023-07-24T16:24:24Z

<p>Mnelson: /* Data Transformation */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST<br /> classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires <br /> the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7086 2023-07-24T16:24:08Z

<p>Mnelson: /* Data Transformation */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Read ==<br /> Reading data requires access to both the dataset, and the structural metadata (DSD) which describes the data in terms of its Dimensions and Concepts. <br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> String dataflow = &quot;dataflow/ECB/BSI/1.0&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> <br /> //Get Structures from URL into an in-memory store<br /> String structureURL = sdmxv2API + &quot;/structure/&quot;+dataflow+&quot;?references=children&quot;;<br /> SdmxBeanRetrievalManager beanRetrieval = new InMemoryRetrievalManagerFast(new ReadableDataLocationTmp(structureURL), new StructureReaderManagerImpl());<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/&quot;+dataflow+&quot;?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> while(dre.moveNextDataset()) {<br /> while(dre.moveNextKeyable()) {<br /> System.out.println(dre.getCurrentKey().getShortCode());<br /> while(dre.moveNextObservation()) {<br /> Observation obs = dre.getCurrentObservation();<br /> System.out.println(obs.getDimensionValue() + &quot;=&quot; + obs.getPrimaryMeasure());<br /> }<br /> } <br /> }<br /> }<br /> <br /> == Data Transformation ==<br /> This example is a bit more involved, as the structures are no longer obtained as a file, but on demand from a web service. This requires the registration of the REST<br /> classes. In addition, as the output data format is SDMX JSON, the writer requires not only access to the DSD, but also all the metadata related to the DSD (Codelists, Concepts). This requires <br /> the SdmxSuperBeanRetrievalManager, which is used to get a fully resolved tree of structural information.<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7085 2023-07-24T16:11:44Z

<p>Mnelson: /* Writing / Transforming Structures */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }<br /> <br /> <br /> == Data Transformation ==<br /> public static void main(String[] args) throws MalformedURLException {<br /> //SDMX WEB SERVICE<br /> String sdmxv2API = &quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2&quot;;<br /> <br /> //Register modules for formats<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> <br /> //Manually register required instances <br /> SingletonStore.registerInstance(new RESTQueryBrokerEngineImpl());<br /> SingletonStore.registerInstance(new StructureQueryBuilderRest());<br /> SingletonStore.registerInstance(new StructureReaderManagerImpl());<br /> <br /> //Set up data reader and writer managers<br /> DataFormatManager dfm = new DataFormatManagerImpl(SdmxJsonDataFormat.INSTANCE, InformationFormatManager.getInstance());<br /> DataReaderManager drm = new DataReaderManagerImpl(dfm);<br /> DataWriterManager dwm = new DataWriterManagerImpl(dfm);<br /> <br /> //JSON output format requires access to the metadata so labels can be resolved<br /> //Wrap the SDMX API ISdmxBeanRestRetrievalManager in a SdmxBeanRetrievalManager so the system can pull back the required structures on demand<br /> ISdmxBeanRestRetrievalManager restBeanRetrievalManager = new RESTSdmxBeanRetrievalManager(sdmxv2API, REST_API_VERSION.v2_0_0);<br /> SdmxBeanRetrievalManager beanRetrieval = new SdmxRestToBeanRetrievalManager(restBeanRetrievalManager);<br /> <br /> //Additionally wrap the SdmxBeanRetrievalManager to create a SdmxSuperBeanRetrievalManager, required when the full 'tree' of structral metadata is <br /> //required (i.e not just the SDMX Data Structure but resolved Concepts and Codelists in one fully resolved Object<br /> SdmxSuperBeanRetrievalManager sbRm = new SdmxSuperBeanRetrievalManagerImpl(beanRetrieval);<br /> <br /> //Register an instance of a SdmxBeanRetreivalContainer with the system, this is used as the default container to get a handle on implementations of the <br /> //structure retrieval interfaces - used by classes including the JSON writers<br /> SingletonStore.registerInstance(new SdmxBeanRetreivalContainer(beanRetrieval, null, sbRm, null, null, restBeanRetrievalManager));<br /> <br /> //Get Data<br /> String dataQueryUrl = sdmxv2API+&quot;/data/dataflow/ECB/BSI/1.0/?c%5BFREQ%5D=A&amp;c%5BREF_AREA%5D=DE,ES&quot;;<br /> ReadableDataLocation sourceData = new ReadableDataLocationTmp(dataQueryUrl);<br /> DataReaderEngine dre = drm.getDataReaderEngine(sourceData, beanRetrieval, null);<br /> <br /> //Create Writer<br /> ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(SdmxJsonDataFormat.INSTANCE, System.out);<br /> <br /> //Transform Data<br /> DataTransformOptions transformOptions = DataTransformOptions.getInstance().setCloseWriter(true); <br /> DataTransformationUtil.copyData(dre, dwe, transformOptions);<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7084 2023-07-24T15:48:39Z

<p>Mnelson: /* Reading Structures */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }<br /> <br /> == Writing / Transforming Structures ==<br /> The same code as Reading Structures, but with the addition lines in BOLD<br /> <br /> * Register the SDMX-JSON module as we are wriring out in this format<br /> * Creates an instance of the StructureWriterManager (typically in an application this would be a singleton). <br /> * Writes the structures in FusionJson format to the system output<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> '''SdmxJsonModule.register();'''<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> '''StructureWriterManager swm = new StructureWriterManagerImpl();'''<br /> '''swm.writeStructures(beans, FusionJsonStructureFormat.getInstance(), System.out);'''<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7083 2023-07-24T15:45:16Z

<p>Mnelson: /* Reading Structures */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Register the SDMX-ML module as the response from the URL is in SDMX-ML format<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> SdmxMLModule.register();<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7082 2023-07-24T15:44:02Z

<p>Mnelson: /* High Level Design */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.<br /> <br /> <br /> = Example Code =<br /> == Reading Structures ==<br /> * Creates an instance of the StructureReaderManager (typically in an application this would be a singleton). <br /> * Creates a wrapper around a URL giving repeated access to the input stream of information, the URL resolves to an SDMX structure file containing mulitple Dataflows<br /> * Convert the stream into SDMX structures<br /> * Loop the Dataflows and print the ID to the console<br /> <br /> public static void main(String[] args) throws MalformedURLException {<br /> StructureReaderManager srm = new StructureReaderManagerImpl();<br /> ReadableDataLocation rdl = new ReadableDataLocationTmp(new URL(&quot;https://demo11.metadatatechnology.com/FusionRegistry/sdmx/v2/structure/dataflow&quot;));<br /> SdmxBeans beans = srm.parseStructures(rdl);<br /> beans.getDataflows().forEach((flow) -&gt; System.out.println(flow.getId()));<br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7081 2023-07-24T15:37:12Z

<p>Mnelson: /* Using Spring/SDMX.IO */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register(); - register schema factories, cross reference resolution, reference metadata reader and writer manager<br /> SdmxMLModule.register(); - register SDMX-XML data and structure format readers and writers<br /> SdmxJsonModule.register(); - register SDMX-JSON data and structure format readers and writers<br /> FusionXLModule.register(); - register FusionXL classes for the excel plugin<br /> SdmxEDIModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> FusionJsonModule.register(); - register Fusion-JSON data and structure format readers and writers<br /> SdmxCsvModule.register(); - register SDMX-CSV data format readers and writers<br /> FusionCoreDataModule.register(); - register SDMX-EDI data and structure format readers and writers<br /> XLSXModule.register(); - register reader and writer for Excel<br /> PublicationTableModule.register(); - classes to support Publication ables<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7080 2023-07-24T13:01:44Z

<p>Mnelson: /* SDMX.IO IoC framework */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register();<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> FusionXLModule.register();<br /> SdmxEDIModule.register();<br /> FusionJsonModule.register();<br /> SdmxCsvModule.register();<br /> FusionCoreDataModule.register();<br /> XLSXModule.register();<br /> PublicationTableModule.register();<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7079 2023-07-24T13:01:22Z

<p>Mnelson: /* Inversion of Control (IoC) */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces and ensuring classes are stateless so that they can be singletons. Managers are typically singletons. This enables the system to be plugged togeather either by using the spring framework, or by using the sdmx.io framework.<br /> <br /> ===Spring Framework===<br /> Spring is used in the Java web application, web.xml references a number of spring bean files, which are located under src/main/resources/spring subfolder of the web application library. The spring beans define the classes and which dependancies they are using. This works well for very large systems such as Fusion Metadata Registry which contains a large number of services and class definitions.<br /> <br /> ===SDMX.IO IoC framework===<br /> In order to remove the dependency on spring, and to simplify simpler projects, Sdmx.io provides a framework which allows instances of singleton classes to be registered to and obtained via the '''SingletonStore''' class. When the class is not a singleton instances can be registered with the '''FusionBeanStore''' class. <br /> <br /> A running application would request a singleton instance in the following way.<br /> <br /> private IActiveDirectorySecuritySettingsManager activeDirectorySecuritySettingsManager = SingletonStore.getSingleton(IActiveDirectorySecuritySettingsManager.class);<br /> <br /> This is typically done at the web service layer, as a web service is constructed outside of the spring framework, after the application has launched.<br /> <br /> There are some use cases where the singleton is not available at the time it is requested, typcially because it is during application startup and the bean has not yet been built and registered. In this scenario the Singleton store can be asked to provide an instance when it is registered.<br /> <br /> SingletonStore.registerInterest(FusionProductInformationRetrievalManager.class, (bean) -&gt; {<br /> infoRetreival = bean;<br /> });<br /> <br /> If the application wants multiple instances of an interface, then the FusionBeanStore is used<br /> <br /> Collection&lt;StructureWriterFactory&gt; allBeans = FusionBeanStore.getBeans(StructureWriterFactory.class);<br /> <br /> ===Using Spring/SDMX.IO===<br /> In order to quickly register modules with the SDMX.IO framework, there are a number of classes that implement '''IFusionModule''', these classes have a register method, which on calling will automatically create and register instances of classes relevent to that module. <br /> <br /> For example:<br /> CoreSDMXModule.register();<br /> SdmxMLModule.register();<br /> SdmxJsonModule.register();<br /> FusionXLModule.register();<br /> SdmxEDIModule.register();<br /> FusionJsonModule.register();<br /> SdmxCsvModule.register();<br /> FusionCoreDataModule.register();<br /> XLSXModule.register();<br /> PublicationTableModule.register();<br /> <br /> The spring framework can be used in conjunction with the SDMX.IO IoC framework, as the class '''SpringBeansContainer''' which is a wrapper around the spring beans contained, can be used by both the SingletonStore and FusionBeansStore. This allows SingletonStore and FusionBeansStore to search both the spring beans container, and any locally registered classes.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7078 2023-07-24T12:37:15Z

<p>Mnelson: /* Java Packages */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods<br /> <br /> The general logical flow of a system is the following<br /> <br /> service --&gt; manager --&gt; engine<br /> --&gt; dao<br /> <br /> The service (web service entry point) is a thin layer which collects the request arguments, with miniumal manipulation, and passes them to the respective Service Interface. The implementation of this Service Interface will create an apporpriate model object (or objects) which contain a consolidated collection of arguements, some validation of arguments can be performed at this point, and then pass these to the manager Interface. The manager will ensure the arguments are valid and the user has permission to perform the action. The manager then provides the workflow required to fulfil the task, by makking calls to engines, daos.<br /> <br /> = High Level Design =<br /> <br /> ==[https://en.wikipedia.org/wiki/Inversion_of_control Inversion of Control] (IoC)==<br /> <br /> IoC is achieved by assigning all implementation classes to Interfaces. Each</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7077 2023-07-24T12:31:07Z

<p>Mnelson: /* Java Packages */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages are as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7076 2023-07-24T12:30:49Z

<p>Mnelson: /* Interface Modules */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> ===Implementation Modules===<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> ===Format Modules===<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML<br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages as as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7075 2023-07-24T12:30:35Z

<p>Mnelson: /* Modules */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> ===Interface Modules===<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> &lt;u&gt;Implementation Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> ===Feature Modules===<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> ===Utility Modules===<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages as as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7074 2023-07-24T12:29:59Z

<p>Mnelson: /* Packages */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> <br /> &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> &lt;u&gt;Implementation Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> &lt;u&gt;Feature Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> &lt;u&gt;Utility Modules&lt;/u&gt;&lt;br/&gt;<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Java Packages ==<br /> Each module is partitioned into Java packages, with the exception of the utility modules, which typically seperate code by its function. <br /> <br /> The generic packages as as follows.<br /> <br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' - contains Java enums and java classes that contain public static constants<br /> '''engine''' - classes that 'do the work', many are stateless singletons, there are also stateful for example DataReader/WriterEngine <br /> '''factory''' - singleton classes that are responisble for generating an engine, typically called by a manager class, factories enable new features to be plugged into the framework<br /> '''filter''' - REST web service filters (defined in web.xml) - filters are called before the web service<br /> '''manager''' - the main entry point for all high level functions, the manager's job is to veryify the request (i.e. a security check) and manage the process of fulfiliing the request by delegating the work to an engine or a DAO, a manager should be a fairly thin layer<br /> '''model''' - data transfer objects <br /> '''service''' (or rest) - web service entry points<br /> '''util''' - utility classes, typically containing public static methods</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7073 2023-07-24T10:09:15Z

<p>Mnelson: /* Architectire */</p> <hr /> <div>= Architectire =<br /> == Modules ==<br /> The sdmx-io modules are as follows<br /> <br /> &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> &lt;u&gt;Implementation Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> &lt;u&gt;Feature Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> &lt;u&gt;Utility Modules&lt;/u&gt;&lt;br/&gt;<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> == Packages ==<br /> <br /> The packages are generally split into the following<br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' <br /> '''engine'''<br /> '''factory'''<br /> '''filter''' <br /> '''manager'''<br /> '''model'''<br /> '''service''' (or rest)<br /> '''util'''<br /> <br /> <br /> <br /> Sdmx Core makes heavy use of Interfaces as opposed to Implementation code, this allows classes to work to interfaces without needing to concern themselves about the actual implementation used.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7072 2023-07-24T10:08:41Z

<p>Mnelson: /* Architectire */</p> <hr /> <div>= Architectire =<br /> The sdmx-io modules are as follows<br /> <br /> &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> &lt;u&gt;Implementation Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> &lt;u&gt;Feature Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - contains generic DAO implementations for storing and querying for objects, not coupled to a specific object store, but we use hibernate, mongodb is also an option<br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> '''fusion-xl''' - exposes web services required by the FusionXL Excel Pluin<br /> <br /> &lt;u&gt;Utility Modules&lt;/u&gt;&lt;br/&gt;<br /> Utility classes, typically public static classes, the utils projects may organise code by function rather then using the generic packaging structure (api, engine, manager, model, ...)<br /> '''fusion-utils''' - generic utility classes, not specific to SDMX<br /> '''fusion-utils-hibernate''' - implementation code to use Hibernate as a means to connect to the DAO<br /> '''fusion-utils-http''' - calling HTTP web services<br /> '''fusion-utils-json''' - JSON related utility classes<br /> '''fusion-utils-mail''' - email utiliity classes<br /> '''fusion-utils-mongodb''' - implementation code to use MongoDB a means to store Objects<br /> '''fusion-utils-sdmx''' - SDMX Specific utilty classes<br /> '''fusion-utils-spring''' - spring related utility classes<br /> '''fusion-utils-xlsx''' - reading/writing Excel workbooks related utility classes<br /> <br /> The packages are generally split into the following<br /> '''api''' - contains interfaces relevent to the module <br /> '''constant''' <br /> '''engine'''<br /> '''factory'''<br /> '''filter''' <br /> '''manager'''<br /> '''model'''<br /> '''service''' (or rest)<br /> '''util'''<br /> <br /> <br /> <br /> Sdmx Core makes heavy use of Interfaces as opposed to Implementation code, this allows classes to work to interfaces without needing to concern themselves about the actual implementation used.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7071 2023-07-24T09:44:01Z

<p>Mnelson: /* Architectire */</p> <hr /> <div>= Architectire =<br /> The sdmx-io modules are as follows<br /> <br /> &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> &lt;u&gt;Implementation Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contain Java classes which are core to SDMX, they implement Interfaces in the higher api modules<br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> '''fusion-core-data''' - contains code to read/write data and generic data related features such as determining series/obs, validation, temporary data stores<br /> '''fusion-core-sdmx''' - contains core code for structural metadata, including temporary store, cross reference resolution, contains some data and reference metadata features <br /> '''fusion-core-structure''' - contains core code for processing structural metadata <br /> '''fusion-core-xlsx''' - contains code to read and write structures and data in Excel <br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> &lt;u&gt;Feature Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules provide feature specific code<br /> '''fusion-core-dao''' - <br /> '''fusion-audit''' - audit events and centralised error handling<br /> '''fusion-sdmx-kafka''' - connecting to, and sending/recieving messages to Apache Kafka<br /> '''fusion-sdmx-publication''' - SDMX Publication Tables: SdmxBean implementations, and buisness logic to materilise table based on definition<br /> '''fusion-sdmx-reporttemplate''' - Reporting Templates: SdmxBean implementations, and buisness logic to genearte excel workbook from template definition, reading data from workbooks<br /> '''fusion-sdmx-rest''' - SDMX web services, and additional web service implementations from APIs defined in fusion-api-service<br /> '''fusion-sdmx-search''' - Lucene search engine for structures<br /> '''fusion-sdmx-security''' - authentication against LDAP/AD/Fusion Security - contains user definition (SecurityDetailsImpl)<br /> <br /> &lt;u&gt;Utility Modules&lt;/u&gt;&lt;br/&gt;<br /> <br /> '''fusion-utils'''<br /> <br /> <br /> The packages are generally split into the following<br /> '''api''' - contains interfaces relevent to the module <br /> '''constant'''<br /> '''engine'''<br /> '''factory'''<br /> '''filter''' <br /> '''manager'''<br /> '''model'''<br /> '''service''' (or rest)<br /> '''util'''<br /> <br /> <br /> <br /> Sdmx Core makes heavy use of Interfaces as opposed to Implementation code, this allows classes to work to interfaces without needing to concern themselves about the actual implementation used.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=DRAFT_SDMX-CORE&diff=7070 2023-07-24T09:12:12Z

<p>Mnelson: Created page with &quot;= Architectire = The sdmx-io modules are as follows &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt; These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses In...&quot;</p> <hr /> <div>= Architectire =<br /> The sdmx-io modules are as follows<br /> <br /> &lt;u&gt;Interface Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java Interfaces only, no Java classes. The sdmx-io framework uses Interfaces over implementation classes in order to achieve loose coupling.<br /> <br /> '''fusion-api''' - contains generic Interfaces that are unrelated to SDMX. Each package in this project is broken down by function, i.e audit, cache, date, email, exception<br /> '''fusion-api-dao''' - contains Interface definitions for database persistence, not coupled to database platform, but all based on storeage and retrieval of Java Objects<br /> '''fusion-api-sdmx''' - contains Interfaces for SDMX Information model (SdmxBeans and related), high level Interfaces for SDMX related functions which cross mulitple modules<br /> '''fusion-api-service''' - contains Interfaces definitions for web services to use - allowing projects the implement web service entry points to not require web service depdenancies<br /> <br /> '''fusion-sdmx-im''' - contains the implementation code for the SDMX Information Model (SdmxBeans and related structures such as Codelists, Concepts, etc)<br /> <br /> &lt;u&gt;Format Modules&lt;/u&gt;&lt;br/&gt;<br /> These modules contains Java classes for reading and writing SDMX Structureal Metadata, Reference Metadata, and Data<br /> <br /> '''fusion-sdmx-csv''' - reading and writing SDMX-CSV format<br /> '''fusion-sdmx-edi''' - reading and writing SDMX-EDI format<br /> '''fusion-sdmx-json''' - reading and writing SDMX-JSON format<br /> '''fusion-sdmx-ml''' reading and writing SDMX-XML <br /> <br /> '''fusion-core-dao'''<br /> '''fusion-core-data'''<br /> '''fusion-core-sdmx'''<br /> '''fusion-audit'''<br /> '''fusion-sdmx-kafka'''<br /> '''fusion-sdmx-publication'''<br /> '''fusion-sdmx-reporttemplate'''<br /> '''fusion-sdmx-rest'''<br /> '''fusion-sdmx-search'''<br /> '''fusion-sdmx-security'''<br /> <br /> '''Utility Classes'''<br /> '''fusion-utils'''<br /> <br /> <br /> The packages are generally split into the following<br /> '''api''' - contains interfaces relevent to the module <br /> '''constant'''<br /> '''engine'''<br /> '''factory'''<br /> '''filter''' <br /> '''manager'''<br /> '''model'''<br /> '''service''' (or rest)<br /> '''util'''<br /> <br /> <br /> <br /> Sdmx Core makes heavy use of Interfaces as opposed to Implementation code, this allows classes to work to interfaces without needing to concern themselves about the actual implementation used.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6955 2023-07-17T16:18:40Z

<p>Mnelson: /* Structure Navigation and Export */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> [[File:FWB Environment Selection.png|thumb|Showing the switch environment options]]<br /> <br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> [[File:FMR Workbench Home.png|thumb|Showing the home page of the Fusion Workbench]]<br /> <br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> https://fwb.sdmxcloud.org/?env=DOT_STAT_STABLE<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> https://fwb.sdmxcloud.org/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=IMF:CS_BOP(1.11)&amp;env=DOT_STAT_STABLE<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Main_Page&diff=6952 2023-07-13T12:56:55Z

<p>Mnelson: /* Using FMR-CL */</p> <hr /> <div>{{DISPLAYTITLE:&lt;span style=&quot;opacity:0;position:absolute;&quot;&gt;{{FULLPAGENAME}}&lt;/span&gt;}}<br /> __NOTOC__<br /> &lt;span style=&quot;font-size:2.0rem;&quot;&gt;Fusion Metadata Registry (FMR) Knowledge Base&lt;/span&gt;<br /> ::[[File:FMR11 screenshot.png|Fusion Metadata Registry|500px]]<br /> ==Change Log and Supported Features==<br /> [[Change Log]] for details of the latest version, bug fixes and updates<br /> <br /> [https://www.sdmx.io/fmr/#features FMR feature list]<br /> <br /> ==Fusion Metadata Registry 11==<br /> Fusion Metadata Registry 11 implements the [https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_5_FINAL-1_0.pdf SDMX 3.0 specification for a structural metadata registry].&lt;br&gt;<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_to_Version_11.4|Upgrading to Version 11.4]] || '''A key change occurs in V11.4, without taking action you may not be able to use FMR'''.<br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_from_Version_10_to_Version_11|Upgrading From Version 10 to Version 11]] || Upgrading from FMR 10 to FMR 11.<br /> |-<br /> | [[FMR 11.0 release notes]] || Key changes in version 11, backward compatibility, breaking changes.<br /> |-<br /> |}<br /> <br /> ====Using Fusion Metadata Registry 11====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To V11|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural_Metadata_V11|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |-<br /> | [[:Category:Reference Metadata|Reference Metadata Management]] || Working with reference metadata using the FMR web user interface and web services API<br /> |}<br /> <br /> ====Use cases====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:SDMX 3.0 Mapping Structures|Mapping]] || Mapping using Structure Maps and Representation Maps<br /> |}<br /> <br /> ====SDMX 3.0 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference_V11|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 3.0 Structures|SDMX 3.0 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> == Fusion Workbench ==<br /> The FMR Workbench (FWB) provides a web user interface (UI) for browsing and maintaining SDMX Structures obtained from external SDMX compliant Structure Web Services, including (and not limited to) - any FMR installation, OECD.STAT, public SDMX web services, such as the SDMX Global Registry,BIS,ECB,ILO,UNICEF.<br /> <br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[FMR_Workbench|FMR Workbench Getting Started]] || Getting started guide for the FMR Workbench<br /> |}<br /> <br /> == FMR-CL ==<br /> '''F'''usion '''M'''etadata '''R'''egistry-'''C'''ommand '''L'''ine (FMR-CL) is a command line application which performs functions against the web services of a running FMR instance.<br /> <br /> ==== Using FMR-CL ====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[Backup and Restore|Structure Backup]] || Supporting backup of structures and transaction history for restore in new installations of FMR<br /> |}<br /> <br /> == FusionXL ==<br /> [https://fmrwiki.sdmxcloud.org/Category:Fusion_XL '''FusionXL''']<br /> FusionXL is a plugin for Microsoft Excel® which connects to the [https://registry.sdmx.org/overview.html Global Registry], Fusion Metadata Registry and Fusion Registry (including the [https://demo.metadatatechnology.com/FusionRegistry/overview.html Metadata Technology demo site]) in order to provide structure browsing, and maintenance.<br /> <br /> ==Fusion Metadata Registry 10==<br /> Fusion Metadata Registry 10 implements the [https://sdmx.org/wp-content/uploads/SDMX_2-1_SECTION_5_RegistrySpecification_2020-07.pdf SDMX 2.1 specification for a structural metadata registry].<br /> ====Using Fusion Metadata Registry 10====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural Metadata|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |}<br /> <br /> ====SDMX 2.1 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference V10|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 2.1 Structures|SDMX 2.1 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> ==About Fusion Metadata Registry==<br /> <br /> ===Key Features===<br /> * SDMX structural metadata registry<br /> * Information model: FMR 10 = SDMX 2.1, FMR 11 = SDMX 3.0<br /> * Formats: SDMX-EDI, SDMX-ML 1.0, 2.0, 2.1, 3.0, SDMX-JSON, SDMX-CSV (data)<br /> * SDMX REST structure API<br /> * Web user interface for browsing, authoring and maintaining structures<br /> <br /> ===Use Cases===<br /> &lt;strong&gt;Structural metadata management&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Centralise and control structural metadata&lt;/li&gt;<br /> &lt;li&gt;Author and maintain structural metadata&lt;/li&gt;<br /> &lt;li&gt;Drive statistics production using centrally-controlled metadata&lt;/li&gt;<br /> &lt;li&gt;Provide an SDMX-compliant a REST structure API&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data collection&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Collect data using metadata-driven Excel forms&lt;/li&gt;<br /> &lt;li&gt;Map (transform) incoming data into standardised structures&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data reporting&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Validate prepared data for compliance with the data collector's specifications prior to reporting&lt;/li&gt;<br /> &lt;li&gt;Map (transform) outgoing data into the collector's required structure&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Statistics production&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Data processing execution engine: data structural validation, mapping to different structures and conversion&lt;/li&gt;<br /> &lt;/ul&gt;</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Main_Page&diff=6951 2023-07-13T12:56:43Z

<p>Mnelson: /* Fusion Workbench */</p> <hr /> <div>{{DISPLAYTITLE:&lt;span style=&quot;opacity:0;position:absolute;&quot;&gt;{{FULLPAGENAME}}&lt;/span&gt;}}<br /> __NOTOC__<br /> &lt;span style=&quot;font-size:2.0rem;&quot;&gt;Fusion Metadata Registry (FMR) Knowledge Base&lt;/span&gt;<br /> ::[[File:FMR11 screenshot.png|Fusion Metadata Registry|500px]]<br /> ==Change Log and Supported Features==<br /> [[Change Log]] for details of the latest version, bug fixes and updates<br /> <br /> [https://www.sdmx.io/fmr/#features FMR feature list]<br /> <br /> ==Fusion Metadata Registry 11==<br /> Fusion Metadata Registry 11 implements the [https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_5_FINAL-1_0.pdf SDMX 3.0 specification for a structural metadata registry].&lt;br&gt;<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_to_Version_11.4|Upgrading to Version 11.4]] || '''A key change occurs in V11.4, without taking action you may not be able to use FMR'''.<br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_from_Version_10_to_Version_11|Upgrading From Version 10 to Version 11]] || Upgrading from FMR 10 to FMR 11.<br /> |-<br /> | [[FMR 11.0 release notes]] || Key changes in version 11, backward compatibility, breaking changes.<br /> |-<br /> |}<br /> <br /> ====Using Fusion Metadata Registry 11====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To V11|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural_Metadata_V11|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |-<br /> | [[:Category:Reference Metadata|Reference Metadata Management]] || Working with reference metadata using the FMR web user interface and web services API<br /> |}<br /> <br /> ====Use cases====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:SDMX 3.0 Mapping Structures|Mapping]] || Mapping using Structure Maps and Representation Maps<br /> |}<br /> <br /> ====SDMX 3.0 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference_V11|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 3.0 Structures|SDMX 3.0 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> == Fusion Workbench ==<br /> The FMR Workbench (FWB) provides a web user interface (UI) for browsing and maintaining SDMX Structures obtained from external SDMX compliant Structure Web Services, including (and not limited to) - any FMR installation, OECD.STAT, public SDMX web services, such as the SDMX Global Registry,BIS,ECB,ILO,UNICEF.<br /> <br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[FMR_Workbench|FMR Workbench Getting Started]] || Getting started guide for the FMR Workbench<br /> |}<br /> <br /> == FMR-CL ==<br /> '''F'''usion '''M'''etadata '''R'''egistry-'''C'''ommand '''L'''ine (FMR-CL) is a command line application which performs functions against the web services of a running FMR instance.<br /> <br /> ==== Using FMR-CL ====<br /> {| <br /> |-<br /> | [[Backup and Restore|Structure Backup]] || Supporting backup of structures and transaction history for restore in new installations of FMR<br /> |}<br /> <br /> == FusionXL ==<br /> [https://fmrwiki.sdmxcloud.org/Category:Fusion_XL '''FusionXL''']<br /> FusionXL is a plugin for Microsoft Excel® which connects to the [https://registry.sdmx.org/overview.html Global Registry], Fusion Metadata Registry and Fusion Registry (including the [https://demo.metadatatechnology.com/FusionRegistry/overview.html Metadata Technology demo site]) in order to provide structure browsing, and maintenance.<br /> <br /> ==Fusion Metadata Registry 10==<br /> Fusion Metadata Registry 10 implements the [https://sdmx.org/wp-content/uploads/SDMX_2-1_SECTION_5_RegistrySpecification_2020-07.pdf SDMX 2.1 specification for a structural metadata registry].<br /> ====Using Fusion Metadata Registry 10====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural Metadata|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |}<br /> <br /> ====SDMX 2.1 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference V10|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 2.1 Structures|SDMX 2.1 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> ==About Fusion Metadata Registry==<br /> <br /> ===Key Features===<br /> * SDMX structural metadata registry<br /> * Information model: FMR 10 = SDMX 2.1, FMR 11 = SDMX 3.0<br /> * Formats: SDMX-EDI, SDMX-ML 1.0, 2.0, 2.1, 3.0, SDMX-JSON, SDMX-CSV (data)<br /> * SDMX REST structure API<br /> * Web user interface for browsing, authoring and maintaining structures<br /> <br /> ===Use Cases===<br /> &lt;strong&gt;Structural metadata management&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Centralise and control structural metadata&lt;/li&gt;<br /> &lt;li&gt;Author and maintain structural metadata&lt;/li&gt;<br /> &lt;li&gt;Drive statistics production using centrally-controlled metadata&lt;/li&gt;<br /> &lt;li&gt;Provide an SDMX-compliant a REST structure API&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data collection&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Collect data using metadata-driven Excel forms&lt;/li&gt;<br /> &lt;li&gt;Map (transform) incoming data into standardised structures&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data reporting&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Validate prepared data for compliance with the data collector's specifications prior to reporting&lt;/li&gt;<br /> &lt;li&gt;Map (transform) outgoing data into the collector's required structure&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Statistics production&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Data processing execution engine: data structural validation, mapping to different structures and conversion&lt;/li&gt;<br /> &lt;/ul&gt;</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Main_Page&diff=6950 2023-07-13T12:56:27Z

<p>Mnelson: /* Fusion Workbench */</p> <hr /> <div>{{DISPLAYTITLE:&lt;span style=&quot;opacity:0;position:absolute;&quot;&gt;{{FULLPAGENAME}}&lt;/span&gt;}}<br /> __NOTOC__<br /> &lt;span style=&quot;font-size:2.0rem;&quot;&gt;Fusion Metadata Registry (FMR) Knowledge Base&lt;/span&gt;<br /> ::[[File:FMR11 screenshot.png|Fusion Metadata Registry|500px]]<br /> ==Change Log and Supported Features==<br /> [[Change Log]] for details of the latest version, bug fixes and updates<br /> <br /> [https://www.sdmx.io/fmr/#features FMR feature list]<br /> <br /> ==Fusion Metadata Registry 11==<br /> Fusion Metadata Registry 11 implements the [https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_5_FINAL-1_0.pdf SDMX 3.0 specification for a structural metadata registry].&lt;br&gt;<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_to_Version_11.4|Upgrading to Version 11.4]] || '''A key change occurs in V11.4, without taking action you may not be able to use FMR'''.<br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_from_Version_10_to_Version_11|Upgrading From Version 10 to Version 11]] || Upgrading from FMR 10 to FMR 11.<br /> |-<br /> | [[FMR 11.0 release notes]] || Key changes in version 11, backward compatibility, breaking changes.<br /> |-<br /> |}<br /> <br /> ====Using Fusion Metadata Registry 11====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To V11|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural_Metadata_V11|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |-<br /> | [[:Category:Reference Metadata|Reference Metadata Management]] || Working with reference metadata using the FMR web user interface and web services API<br /> |}<br /> <br /> ====Use cases====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:SDMX 3.0 Mapping Structures|Mapping]] || Mapping using Structure Maps and Representation Maps<br /> |}<br /> <br /> ====SDMX 3.0 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference_V11|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 3.0 Structures|SDMX 3.0 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> == Fusion Workbench ==<br /> The FMR Workbench (FWB) provides a web user interface (UI) for browsing and maintaining SDMX Structures obtained from external SDMX compliant Structure Web Services, including (and not limited to) - any FMR installation, OECD.STAT, public SDMX web services, such as the SDMX Global Registry,BIS,ECB,ILO,UNICEF.<br /> <br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; [[FMR_Workbench|FMR Workbench Getting Started]] || Getting started guide for the FMR Workbench<br /> |}<br /> <br /> == FMR-CL ==<br /> '''F'''usion '''M'''etadata '''R'''egistry-'''C'''ommand '''L'''ine (FMR-CL) is a command line application which performs functions against the web services of a running FMR instance.<br /> <br /> ==== Using FMR-CL ====<br /> {| <br /> |-<br /> | [[Backup and Restore|Structure Backup]] || Supporting backup of structures and transaction history for restore in new installations of FMR<br /> |}<br /> <br /> == FusionXL ==<br /> [https://fmrwiki.sdmxcloud.org/Category:Fusion_XL '''FusionXL''']<br /> FusionXL is a plugin for Microsoft Excel® which connects to the [https://registry.sdmx.org/overview.html Global Registry], Fusion Metadata Registry and Fusion Registry (including the [https://demo.metadatatechnology.com/FusionRegistry/overview.html Metadata Technology demo site]) in order to provide structure browsing, and maintenance.<br /> <br /> ==Fusion Metadata Registry 10==<br /> Fusion Metadata Registry 10 implements the [https://sdmx.org/wp-content/uploads/SDMX_2-1_SECTION_5_RegistrySpecification_2020-07.pdf SDMX 2.1 specification for a structural metadata registry].<br /> ====Using Fusion Metadata Registry 10====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural Metadata|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |}<br /> <br /> ====SDMX 2.1 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference V10|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 2.1 Structures|SDMX 2.1 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> ==About Fusion Metadata Registry==<br /> <br /> ===Key Features===<br /> * SDMX structural metadata registry<br /> * Information model: FMR 10 = SDMX 2.1, FMR 11 = SDMX 3.0<br /> * Formats: SDMX-EDI, SDMX-ML 1.0, 2.0, 2.1, 3.0, SDMX-JSON, SDMX-CSV (data)<br /> * SDMX REST structure API<br /> * Web user interface for browsing, authoring and maintaining structures<br /> <br /> ===Use Cases===<br /> &lt;strong&gt;Structural metadata management&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Centralise and control structural metadata&lt;/li&gt;<br /> &lt;li&gt;Author and maintain structural metadata&lt;/li&gt;<br /> &lt;li&gt;Drive statistics production using centrally-controlled metadata&lt;/li&gt;<br /> &lt;li&gt;Provide an SDMX-compliant a REST structure API&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data collection&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Collect data using metadata-driven Excel forms&lt;/li&gt;<br /> &lt;li&gt;Map (transform) incoming data into standardised structures&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data reporting&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Validate prepared data for compliance with the data collector's specifications prior to reporting&lt;/li&gt;<br /> &lt;li&gt;Map (transform) outgoing data into the collector's required structure&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Statistics production&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Data processing execution engine: data structural validation, mapping to different structures and conversion&lt;/li&gt;<br /> &lt;/ul&gt;</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Main_Page&diff=6949 2023-07-13T12:55:44Z

<p>Mnelson: /* FMR-CL */</p> <hr /> <div>{{DISPLAYTITLE:&lt;span style=&quot;opacity:0;position:absolute;&quot;&gt;{{FULLPAGENAME}}&lt;/span&gt;}}<br /> __NOTOC__<br /> &lt;span style=&quot;font-size:2.0rem;&quot;&gt;Fusion Metadata Registry (FMR) Knowledge Base&lt;/span&gt;<br /> ::[[File:FMR11 screenshot.png|Fusion Metadata Registry|500px]]<br /> ==Change Log and Supported Features==<br /> [[Change Log]] for details of the latest version, bug fixes and updates<br /> <br /> [https://www.sdmx.io/fmr/#features FMR feature list]<br /> <br /> ==Fusion Metadata Registry 11==<br /> Fusion Metadata Registry 11 implements the [https://sdmx.org/wp-content/uploads/SDMX_3-0-0_SECTION_5_FINAL-1_0.pdf SDMX 3.0 specification for a structural metadata registry].&lt;br&gt;<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_to_Version_11.4|Upgrading to Version 11.4]] || '''A key change occurs in V11.4, without taking action you may not be able to use FMR'''.<br /> |-<br /> | style=&quot;width:300px&quot; | [[Upgrading_from_Version_10_to_Version_11|Upgrading From Version 10 to Version 11]] || Upgrading from FMR 10 to FMR 11.<br /> |-<br /> | [[FMR 11.0 release notes]] || Key changes in version 11, backward compatibility, breaking changes.<br /> |-<br /> |}<br /> <br /> ====Using Fusion Metadata Registry 11====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To V11|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural_Metadata_V11|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |-<br /> | [[:Category:Reference Metadata|Reference Metadata Management]] || Working with reference metadata using the FMR web user interface and web services API<br /> |}<br /> <br /> ====Use cases====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:SDMX 3.0 Mapping Structures|Mapping]] || Mapping using Structure Maps and Representation Maps<br /> |}<br /> <br /> ====SDMX 3.0 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference_V11|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 3.0 Structures|SDMX 3.0 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> == Fusion Workbench ==<br /> The FMR Workbench (FWB) provides a web user interface (UI) for browsing and maintaining SDMX Structures obtained from external SDMX compliant Structure Web Services, including (and not limited to) - any FMR installation, OECD.STAT, public SDMX web services, such as the SDMX Global Registry,BIS,ECB,ILO,UNICEF.<br /> <br /> {| <br /> |-<br /> | [[FMR_Workbench|FMR Workbench Getting Started]] || Getting started guide for the FMR Workbench<br /> |}<br /> <br /> <br /> == FMR-CL ==<br /> '''F'''usion '''M'''etadata '''R'''egistry-'''C'''ommand '''L'''ine (FMR-CL) is a command line application which performs functions against the web services of a running FMR instance.<br /> <br /> ==== Using FMR-CL ====<br /> {| <br /> |-<br /> | [[Backup and Restore|Structure Backup]] || Supporting backup of structures and transaction history for restore in new installations of FMR<br /> |}<br /> <br /> == FusionXL ==<br /> [https://fmrwiki.sdmxcloud.org/Category:Fusion_XL '''FusionXL''']<br /> FusionXL is a plugin for Microsoft Excel® which connects to the [https://registry.sdmx.org/overview.html Global Registry], Fusion Metadata Registry and Fusion Registry (including the [https://demo.metadatatechnology.com/FusionRegistry/overview.html Metadata Technology demo site]) in order to provide structure browsing, and maintenance.<br /> <br /> ==Fusion Metadata Registry 10==<br /> Fusion Metadata Registry 10 implements the [https://sdmx.org/wp-content/uploads/SDMX_2-1_SECTION_5_RegistrySpecification_2020-07.pdf SDMX 2.1 specification for a structural metadata registry].<br /> ====Using Fusion Metadata Registry 10====<br /> {| <br /> |-<br /> | [[Quick_Start_Guide|Quick Start Guide]] || Install and configure a basic FMR installation<br /> |- <br /> | style=&quot;width:300px&quot; | [[:Category:How_To|How to Guides]]|| Guidance on specific tasks using FMR<br /> |-<br /> | [[:Category:FMR_Configuration_Reference|Configuration Reference]]|| Configuring FMR<br /> |-<br /> | [[:Category:Structural Metadata|Structural Metadata Management]] || Creating and maintaining SDMX structures using the FMR web user interface<br /> |}<br /> <br /> ====SDMX 2.1 Theory====<br /> {| <br /> |-<br /> | style=&quot;width:300px&quot; | [[:Category:Concepts_Reference V10|Concepts Reference]] || Explanation of key concepts and terms used in SDMX<br /> |-<br /> | [[:Category:SDMX 2.1 Structures|SDMX 2.1 Structures Reference]] || Details on each of the key 'maintainable' and 'identifiable' SDMX structures<br /> |-<br /> | [[:Category:FMR_Formats_Reference|Formats Reference]] || Details on each of the data and structure formats available in Fusion Metadata Registry including XML, JSON and CSV<br /> |-<br /> | [[:Category:FMR_REST_API_Reference|REST API Web Services Reference]] || Purpose, parameters, request content and response content for each of the published web services together with examples<br /> |}<br /> <br /> ==About Fusion Metadata Registry==<br /> <br /> ===Key Features===<br /> * SDMX structural metadata registry<br /> * Information model: FMR 10 = SDMX 2.1, FMR 11 = SDMX 3.0<br /> * Formats: SDMX-EDI, SDMX-ML 1.0, 2.0, 2.1, 3.0, SDMX-JSON, SDMX-CSV (data)<br /> * SDMX REST structure API<br /> * Web user interface for browsing, authoring and maintaining structures<br /> <br /> ===Use Cases===<br /> &lt;strong&gt;Structural metadata management&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Centralise and control structural metadata&lt;/li&gt;<br /> &lt;li&gt;Author and maintain structural metadata&lt;/li&gt;<br /> &lt;li&gt;Drive statistics production using centrally-controlled metadata&lt;/li&gt;<br /> &lt;li&gt;Provide an SDMX-compliant a REST structure API&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data collection&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Collect data using metadata-driven Excel forms&lt;/li&gt;<br /> &lt;li&gt;Map (transform) incoming data into standardised structures&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Data reporting&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Validate prepared data for compliance with the data collector's specifications prior to reporting&lt;/li&gt;<br /> &lt;li&gt;Map (transform) outgoing data into the collector's required structure&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;strong&gt;Statistics production&lt;/strong&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;Data processing execution engine: data structural validation, mapping to different structures and conversion&lt;/li&gt;<br /> &lt;/ul&gt;</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6947 2023-07-13T12:51:12Z

<p>Mnelson: Mnelson moved page DRAFT-FMR Workbench to FMR Workbench</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> [[File:FWB Environment Selection.png|thumb|Showing the switch environment options]]<br /> <br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> [[File:FMR Workbench Home.png|thumb|Showing the home page of the Fusion Workbench]]<br /> <br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6946 2023-07-13T12:50:28Z

<p>Mnelson: /* Apache Tomcat Example */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> [[File:FWB Environment Selection.png|thumb|Showing the switch environment options]]<br /> <br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> [[File:FMR Workbench Home.png|thumb|Showing the home page of the Fusion Workbench]]<br /> <br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6945 2023-07-13T12:50:20Z

<p>Mnelson: /* Apache Tomcat Example */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> [[File:FWB Environment Selection.png|thumb|Showing the switch environment options]]<br /> <br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> [[File:FMR Workbench Home.png|center|Showing the home page of the Fusion Workbench]]<br /> <br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6944 2023-07-13T12:49:49Z

<p>Mnelson: /* Configuration File Properties */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> [[File:FWB Environment Selection.png|thumb|Showing the switch environment options]]<br /> <br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> [[File:FMR Workbench Home.png|center]]<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6943 2023-07-13T12:49:11Z

<p>Mnelson: /* Apache Tomcat Example */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> [[File:FWB Environment Selection.png|center]]<br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> [[File:FMR Workbench Home.png|center]]<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6942 2023-07-13T12:48:46Z

<p>Mnelson: /* Apache Tomcat Example */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> [[File:FWB Environment Selection.png|center]]<br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> [[File:FMR Workbench Home.png|thumb]]<br /> <br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6941 2023-07-13T12:48:25Z

<p>Mnelson: /* Configuration File Properties */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> [[File:FWB Environment Selection.png|center]]<br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). &lt;br/&gt;<br /> Options include:&lt;br/&gt;&lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> |}<br /> <br /> == Apache Tomcat Example ==<br /> If using Apache Tomcat, the war file would be placed under the ‘&lt;tomcat&gt;\webapps’ folder. <br /> <br /> The name of the war file will be the name of the Web Application when accessed in the web browser, for example if the war file is called fwb.war, the application would be accessed using the following URL syntax:<br /> <br /> http(s)://[server]:[port]/'''fwb'''<br /> <br /> Apache Tomcat can be started using the '''bin\startup.bat''' or on unix '''bin\startup.sh script'''. An alternative is to start Tomcat as a windows service.<br /> <br /> On Tomcat startup the war file will be unpacked, creating a folder of the same name in the &lt;tomcat&gt;\webapps folder.<br /> <br /> After Tomcat has started the Fusion Workbench would be accessed from the following URL.<br /> <br /> http://localhost:8080/fwb<br /> <br /> This URL assumes the following conditions are true:<br /> # The web browser is running on the same machine as the Tomcat server ('''localhost''' refers to the local machine).<br /> # No configuration changes have been made to the official distribution of Apache Tomcat (default port is '''8080''').<br /> # The FWB distribution has the file name fwb.war (fwb.war resolves to '''/fwb''' in the URL path)<br /> <br /> Assuming the configuration files are in the correct folder and are valid JSON with no missing required properties, the URL will resolve to the UI of the FWB.<br /> <br /> [[File:FMR Workbench Home.png|thumb]]<br /> <br /> == Error Resolution ==<br /> If the home page of the FWB fails to load, please check the log files from Apache Tomcat. Other than the Apache Tomcat log files, the FWB will create 2 application specific log files:<br /> # FMRWorkbench.log – this file contains all logging output from the FWB. Logs are output if they have a level of INFO or higher.<br /> # FMRWorkbenchErrors.log – this file contains a subset of the logs from FMRWorkbench.log. It only contains logs that were written with a severity of WARN or higher.<br /> <br /> To change the logging level, edit the '''logback.xml''' file which is found under the following location in Tomcat<br /> <br /> &lt;tomcat home&gt;\webapps\fwb\WEB-INF\classes\logback.xml<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=File:FMR_Workbench_Home.png&diff=6940 2023-07-13T12:47:37Z

<p>Mnelson: </p> <hr /> <div>FMR Workbench Home</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6939 2023-07-13T12:44:39Z

<p>Mnelson: /* Overview */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Getting Started = <br /> The FMR Workbench (FWB) is deployed as a Web Application Archive (war) file. The war file must be deployed to a Java Servlet container such as Apache Tomcat. The Servlet container will host the application so that it can be accessed over HTTP as a web application.<br /> <br /> == Configuration File Location == <br /> The FMR Workbench does not use a database, instead it obtains its configuration settings from a configuration file called fwb_config.json.<br /> <br /> An example configuration file is provided with the software. The configuration file needs to be placed in the following directory (the directory must be created if it does not exist):<br /> <br /> &lt;USER HOME&gt;\MetadataTechnology\FusionRegistry<br /> <br /> Example:<br /> C:\Users\Matthew Nelson\MetadataTechnology\FusionRegistry<br /> <br /> == Configuration File Properties ==<br /> The configuration file is split into 2 parts:<br /> <br /> # Server Properties<br /> # Environment Definitions<br /> <br /> {<br /> &quot;Properties&quot;: { &lt;span style=&quot;color:#C00000&quot;&gt;server properties go here&lt;/span&gt; },<br /> &quot;Environments&quot;: [ &lt;span style=&quot;color:#C00000&quot;&gt;environment definitions go here&lt;/span&gt; ]<br /> }<br /> <br /> '''Properties'''&lt;br/&gt;<br /> There is only 1 property which is required for the FWB, this is the URL to the FWB application. This property enables the application to know where to find required resources, and how to redirect users to the correct pages. <br /> <br /> &quot;Properties&quot;: {<br /> server.url: &quot;http://localhost:8080/fwb&quot;<br /> }<br /> <br /> Localhost can be used for testing purposes when deploying the application to the same computer that the web browser is run from. For a live deployment this URL must be changed accordingly.<br /> <br /> '''Environments'''&lt;br/&gt;<br /> The Environment property has an array of JSON Objects. Each JSON Object in this array defines a distinct Environment which the FWB can switch to in order to browse and optionally create/maintain the SDMX Structures of that Environment. <br /> <br /> [[File:FWB Environment Selection.png|center]]<br /> <br /> An Environment consists of an id and a name and defines the web service entry points to retrieve, save, and delete structures. The retrieve web service is required, the save and delete entry points are optional. <br /> <br /> The Environment definition also contains details on the authentication mechanism used to access the contents of the web services.<br /> <br /> An example of an Environment definition is given below:<br /> {<br /> &quot;id&quot;: &quot;DOT_STAT&quot;,<br /> &quot;name&quot; : &quot;Dot Stat Test&quot;,<br /> &quot;api&quot;: {<br /> &quot;query&quot;: &quot;https://nsi-demo-reset.siscc.org/rest&quot;,<br /> &quot;persist&quot;: &quot;https://nsi-demo-reset.siscc.org/rest/structure&quot;,<br /> &quot;delete&quot;: &quot;https://nsi-demo-stable.siscc.org/rest/structure&quot;,<br /> &quot;version&quot;: &quot;1.5.0&quot;<br /> },<br /> &quot;agencies&quot;: [&quot;SDMX&quot;, &quot;METATECH&quot;],<br /> &quot;auth&quot;: &quot;OIDC_PKCE&quot;,<br /> &quot;locale&quot;: [&quot;en&quot;, &quot;fr&quot;],<br /> &quot;oidc&quot;: {<br /> &quot;config&quot;: &quot;https://keycloak.siscc.org/auth/realms/OECD/.well-known/openid-configuration&quot;,<br /> &quot;clientId&quot;: &quot;app&quot;<br /> },<br /> &quot;format&quot;: &quot;mlv2.1&quot;<br /> }<br /> <br /> The following table describes the properties for an Environment definition based on the above example:<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Property !! Description<br /> |-<br /> | '''id''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; This is the ID of the Environment. It is used in the REST API to switch environments<br /> |-<br /> | '''name''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; The name of the Environment as presented to the user<br /> |-<br /> | '''api''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; contains the base SDMX REST URL for each web service of the Environment<br /> |-<br /> | '''api.query''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; structure query web service<br /> |-<br /> | '''api.persist''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support add/edit functionality. HTTP POST messages will be sent to this URL<br /> |-<br /> | '''api.delete''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; to support delete functionality, HTTP DELETE messages will be sent to this URL<br /> |-<br /> | '''api.version''' || &lt;span style=&quot;color:#C00000&gt;'''Required'''&lt;/span&gt; the Version of the SDMX REST API. Supported values are 1.5.0 and 2.0.0<br /> |-<br /> | '''api.timeout''' || &lt;span style=&quot;color:#4472C4&gt;'''Optional'''&lt;/span&gt; maximum number of seconds to wait for response from API before timing out. Defaults to120 seconds.<br /> |-<br /> | '''format''' || The SDMX structure format used when POSTing structures to web service (for the use case of structure persistence). <br /> Options include:<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv2.1'''&lt;/span&gt; – XML format SDMX-ML v.2.1 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''mlv3.0'''&lt;/span&gt; – XML format SDMX-ML v.3.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv1'''&lt;/span&gt; – JSON format SDMX-JSON v.1.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''jsonv2'''&lt;/span&gt; – JSON format SDMX-JSON v.2.0 &lt;br/&gt;<br /> &lt;span style=&quot;color:#4472C4&gt;'''fusion'''&lt;/span&gt; – the FMR specific format. This format should be used if connected to another FMR instance as it covers both SDMX structures and non-SDMX structures such as Publication Tables and Reporting Templates &lt;br/&gt;<br /> <br /> |}<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=File:FWB_Environment_Selection.png&diff=6938 2023-07-13T12:36:33Z

<p>Mnelson: </p> <hr /> <div>FWB Environment Selection</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6937 2023-07-13T12:30:53Z

<p>Mnelson: /* Overview */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The '''F'''MR '''W'''ork'''B'''ench (FWB) is a User Interface (UI) for browsing and maintaining SDMX Structures obtained from SDMX compliant Structure Web Services. The UI of the FWB is a stripped-down version of the Fusion Metadata Registry (FMR) User Interface; it is stripped down because the UI of the FWB supports the use case of viewing and maintaining structural metadata, whilst the FMR supports additional use cases including [[Validate_data|data validation]], [[Convert_data_between_SDMX_formats_V11|data conversion]], and has administrative UI pages. Whilst the FMR uses a database to store its configuration settings, the FWB configuration is stored in a file. '''The FWB has no database requirements'''.<br /> <br /> The FWB UI browses and displays the SDMX structures of the '''Environment''' it is connected to. An Environment is a connection to a collection of related SDMX web services for a single SDMX service offering. Related web services include the web service entry point to retrieve the structures, the entry point to save structures, and the entry point to delete structures.<br /> <br /> The UI of the FWB enables the user to switch their connection to an alternative Environment; the list of available Environments is defined in the configuration file for the FWB. The connected Environment is associated with the current user; this enables multiple users of the FWB to be connected to different Environments simultaneously.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6845 2023-02-14T12:04:27Z

<p>Mnelson: /* Structure Navigation and Export */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The FMR Workbench is a web application which provides the web user interface to view and maintain structural metadata, the critical difference between the FMR Workbench and the Fusion Metadata Registry is that the FMR Workbench does not have a database - the structures that it views and maintains come from a separate web service.<br /> <br /> The FMR Workbench reads a config file on startup which describes one or more 'services'. Each service is an SDMX endpoint with a query API and an optional persistence API. The web user interface of the FMR Workbench enables users to switch between services. The user can then browse the SDMX Structures of the service they are connected to.<br /> <br /> The connected service is attached to the user's session, therefore it is possible for multiple users to use the same FMR-Workbench, with each user connecting to a different service.<br /> <br /> The FMR-Workbench can be used to edit, and save structures back to the service, if the service supports persistence, and the user is logged in.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6844 2023-02-14T12:04:17Z

<p>Mnelson: /* Structure Navigation and Export */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The FMR Workbench is a web application which provides the web user interface to view and maintain structural metadata, the critical difference between the FMR Workbench and the Fusion Metadata Registry is that the FMR Workbench does not have a database - the structures that it views and maintains come from a separate web service.<br /> <br /> The FMR Workbench reads a config file on startup which describes one or more 'services'. Each service is an SDMX endpoint with a query API and an optional persistence API. The web user interface of the FMR Workbench enables users to switch between services. The user can then browse the SDMX Structures of the service they are connected to.<br /> <br /> The connected service is attached to the user's session, therefore it is possible for multiple users to use the same FMR-Workbench, with each user connecting to a different service.<br /> <br /> The FMR-Workbench can be used to edit, and save structures back to the service, if the service supports persistence, and the user is logged in.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6843 2023-02-14T12:03:57Z

<p>Mnelson: /* Structure Navigation and Export */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The FMR Workbench is a web application which provides the web user interface to view and maintain structural metadata, the critical difference between the FMR Workbench and the Fusion Metadata Registry is that the FMR Workbench does not have a database - the structures that it views and maintains come from a separate web service.<br /> <br /> The FMR Workbench reads a config file on startup which describes one or more 'services'. Each service is an SDMX endpoint with a query API and an optional persistence API. The web user interface of the FMR Workbench enables users to switch between services. The user can then browse the SDMX Structures of the service they are connected to.<br /> <br /> The connected service is attached to the user's session, therefore it is possible for multiple users to use the same FMR-Workbench, with each user connecting to a different service.<br /> <br /> The FMR-Workbench can be used to edit, and save structures back to the service, if the service supports persistence, and the user is logged in.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6842 2023-02-14T12:03:50Z

<p>Mnelson: /* Structure Navigation and Export */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The FMR Workbench is a web application which provides the web user interface to view and maintain structural metadata, the critical difference between the FMR Workbench and the Fusion Metadata Registry is that the FMR Workbench does not have a database - the structures that it views and maintains come from a separate web service.<br /> <br /> The FMR Workbench reads a config file on startup which describes one or more 'services'. Each service is an SDMX endpoint with a query API and an optional persistence API. The web user interface of the FMR Workbench enables users to switch between services. The user can then browse the SDMX Structures of the service they are connected to.<br /> <br /> The connected service is attached to the user's session, therefore it is possible for multiple users to use the same FMR-Workbench, with each user connecting to a different service.<br /> <br /> The FMR-Workbench can be used to edit, and save structures back to the service, if the service supports persistence, and the user is logged in.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> It is possible to automatically link to a page in the Fusion Registry against a specific service by providing the '''env''' argument in the URL, for example:<br /> <br /> <br /> http://localhost:8080/FMR-WB/items/conceptscheme.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to a specific Concept Scheme in the User Interface, from the DOT_STAT environment.<br /> <br /> http://localhost:8080/items/conceptscheme-wizard.html?urn=urn:sdmx:org.sdmx.infomodel.conceptscheme.ConceptScheme=OECD:CS_TOURISM(1.0)&amp;env=DOT_STAT<br /> <br /> The above link will take the user to the Concept Scheme Wizard (the edit pages for a Concept Scheme). The Concept Scheme wizard will be editing the CS_TOURISM concept in the DOT_STAT Environment. As this page is a secure resource, the user will be automatically redirected to the login page if they are not already logged into the service.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Data_Validation_Web_Service&diff=6800 2022-12-21T12:49:03Z

<p>Mnelson: /* HTTP Headers */</p> <hr /> <div>[[Category:FMR REST API Reference]]<br /> = Overview =<br /> {| class=&quot;wikitable&quot;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Entry Point&lt;/b&gt;|| &lt;b&gt; /ws/public/data/validate &lt;/b&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Access&lt;/b&gt;|| &lt;span style='color:green'&gt;&lt;b&gt;Public&lt;/b&gt;&lt;/span&gt; (default). Configurable to Private<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Http Method&lt;/b&gt;|| POST<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accepts&lt;/b&gt;|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Compression&lt;/b&gt; || Zip files supported, if loading from URL gzip responses supported<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Content-Type&lt;/b&gt;|| &lt;p&gt;1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile&lt;/p&gt;<br /> &lt;p&gt;2. application/text or application/xml (if submitting data in the body of the POST)&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Format&lt;/b&gt;|| application/json<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Statuses&lt;/b&gt;|| &lt;p&gt;&lt;b&gt;200&lt;/b&gt; - Validation could be performed&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;400&lt;/b&gt; - Validation could not be performed (either an unreadable dataset, or unresolvable reference to a required structure)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;401&lt;/b&gt; - Unauthorized (if access has been restricted)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;500&lt;/b&gt; - Server Error&lt;/p&gt;<br /> |}<br /> <br /> = HTTP Headers =<br /> {| class=&quot;wikitable&quot;<br /> |-<br /> !|HTTP Header || Purpose || Allowed Values<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Data-Format&lt;/b&gt;|| Used to inform the server when the data is in CSV format. ||csv;delimiter=[delimiter]<br /> &lt;p&gt;Where [delimiter] is either:&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;comma&lt;/li&gt;<br /> &lt;li&gt;tab&lt;/li&gt;<br /> &lt;li&gt;semicolon&lt;/li&gt;<br /> &lt;li&gt;space&lt;/li&gt;<br /> &lt;/ul&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Structure&lt;/b&gt; || &lt;p&gt;(optional) Provides the structure to validate the data against.&lt;/p&gt;<br /> <br /> &lt;p&gt;This information may be present in the header of the dataset, if provided this value will override the value in the dataset.&lt;/p&gt;<br /> &lt;p&gt;Fusion Metadada Registry v10.4.7 onwards accepts a comma separated list of URNs if multiple datasets exist in the loaded file, however if multiple URNs are provided, URN must be of a Dataflow or Provision Agreement and the datasets must identify at a minimum the Data Structure that is needed to read the file.&lt;/p&gt;<br /> <br /> || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Metrics&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||<br /> &lt;p&gt;Optional. Includes metrics on the validation.&lt;/p&gt;<br /> &lt;p&gt;This will add extra detail to the validation report.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Valid&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;Optional. Instructs the service to include a dataset with all the valid series and observations in the response.&lt;/p&gt; <br /> <br /> &lt;p&gt;As the result will contain a separate file for the dataset, the response format will be set to either multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.&lt;/p&gt;<br /> &lt;p&gt;The file is called ValidData with the file extension based on the output format.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Invalid&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;<br /> Optional. Instructs the service to include a dataset with all the invalid series and observations in the response. &lt;/p&gt;<br /> <br /> &lt;p&gt;As the result will contain a separate file for the dataset, the response format will be set to either multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.&lt;/p&gt;<br /> &lt;p&gt;The file is called InvalidData with the file extension based on the output format.&lt;/p&gt;<br /> <br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accept&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;Optional. Instructs the service which data output format to output the valid or invalid datasets in.<br /> &lt;/p&gt;<br /> &lt;p&gt;'''Note''': From FMR 11.5.0 the format (if not specified) defaults to the input format. Previous versions defaulted to SDMX Structure Specific 2.1&lt;/p&gt;<br /> &lt;p&gt;'''Note''': This Header is only used if Inc-Valid or Inc-Invalid are set to true.&lt;/p&gt;<br /> ||[[Data_Formats|See Accept Formats]]<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Zip&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||Optional. Compresses the output as a zip file. If used in conjunction with &lt;b&gt;Inc-Valid&lt;/b&gt; or &lt;b&gt;Inc-Invalid&lt;/b&gt; the zip will contain multiple files.<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Prior-Data-Dependent&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;<br /> Optional. This allows data to be validated under the assumption that other data will provide missing information. If this value is set to true, particular data validators will not be used when validating the data. These validators are &quot;Mandatory Observations&quot; and &quot;Valid Calculations&quot;.&lt;/p&gt;<br /> <br /> &lt;p&gt;Default value is false.&lt;/p&gt;<br /> <br /> ||Boolean (true/false)<br /> |}<br /> <br /> = Response =<br /> &lt;p&gt;<br /> The validation output contains both human readable error descriptions, as well as machine processible locations of the errors within the dataset. The location in the dataset is described as a key or observation locator in the format; A:UK:M:2008 – where each component relates to the Dimension value, separated by a colon. If the error position is observation, the last part of the key is the observation time period.&lt;/p&gt;<br /> <br /> &lt;p&gt;There are 3 types of output that can be produced which share a common structure: unable to parse input(returns HTTP 400); able to parse input but references invalid data structure (returns HTTP 200); parsed input and returns output, which may have validation errors (return HTTPS 200). Below are examples of each:&lt;/p&gt;<br /> <br /> == Valid Dataset ==<br /> <br /> {<br /> &quot;Meta&quot;: {<br /> &quot;RequestTime&quot;: 1564410081711,<br /> &quot;Duration&quot;: 43<br /> },<br /> &quot;FileFormat&quot;: &quot;Structure Specific (Compact) v2.1&quot;,<br /> &quot;Prepared&quot;: &quot;2019-07-29T10:23:01&quot;,<br /> &quot;SenderId&quot;: &quot;FR_DEMO&quot;,<br /> &quot;DataSetId&quot;: null,<br /> &quot;Status&quot;: &quot;Complete&quot;,<br /> &quot;Errors&quot;: false,<br /> &quot;Datasets&quot;: [<br /> {<br /> &quot;DSD&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=OECD:HIGH_AGLINK_2011(1.0)&quot;,<br /> &quot;Dataflow&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:AGRIC_OUTLOOK_2011_2020(1.0)&quot;,<br /> &quot;DataProvider&quot;: &quot;urn:sdmx:org.sdmx.infomodel.base.DataProvider=METATECH:DATA_PROVIDERS(1.0).METATECH&quot;,<br /> &quot;ProvisionAgreement&quot;: &quot;urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)&quot;,<br /> &quot;KeysCount&quot;: 2,<br /> &quot;ObsCount&quot;: 62,<br /> &quot;GroupsCount&quot;: 0,<br /> &quot;Errors&quot;: false<br /> &quot;ReportedPeriods&quot;: {<br /> &quot;A&quot;: {<br /> &quot;Name&quot;: &quot;Annual&quot;,<br /> &quot;StartPeriod&quot;: &quot;1990&quot;,<br /> &quot;EndPeriod&quot;: &quot;2020&quot;<br /> }<br /> }, <br /> }<br /> ],<br /> &quot;PreventsConversion&quot;: false,<br /> &quot;PreventsPublication&quot;: false<br /> }<br /> <br /> <br /> == Dataset with Errors ==<br /> {<br /> &quot;Meta&quot;: {<br /> &quot;RequestTime&quot;: 1564401209760,<br /> &quot;Duration&quot;: 34<br /> },<br /> &quot;InvalidData&quot;: {<br /> &quot;Datasets&quot;: [<br /> {<br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)&quot;,<br /> &quot;Series&quot;: 2,<br /> &quot;Observations&quot;: 61,<br /> &quot;Groups&quot;: 0<br /> }<br /> ]<br /> },<br /> &quot;ValidData&quot;: {<br /> &quot;Datasets&quot;: [<br /> {<br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)&quot;,<br /> &quot;Series&quot;: 2,<br /> &quot;Observations&quot;: 32,<br /> &quot;Groups&quot;: 0<br /> }<br /> ]<br /> },<br /> &quot;FileFormat&quot;: &quot;Structure Specific (Compact) v2.1&quot;,<br /> &quot;Prepared&quot;: &quot;2019-07-29T10:23:01&quot;,<br /> &quot;SenderId&quot;: &quot;FR_DEMO&quot;,<br /> &quot;DataSetId&quot;: null,<br /> &quot;Status&quot;: &quot;Complete&quot;,<br /> &quot;Errors&quot;: true,<br /> &quot;Datasets&quot;: [<br /> {<br /> &quot;DSD&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=OECD:HIGH_AGLINK_2011(1.0)&quot;,<br /> &quot;Dataflow&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:AGRIC_OUTLOOK_2011_2020(1.0)&quot;,<br /> &quot;DataProvider&quot;: &quot;urn:sdmx:org.sdmx.infomodel.base.DataProvider=METATECH:DATA_PROVIDERS(1.0).METATECH&quot;,<br /> &quot;ProvisionAgreement&quot;: &quot;urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=OECD:OECD_AGRIC_OUTLOOK(1.0)&quot;,<br /> &quot;KeysCount&quot;: 3,<br /> &quot;ObsCount&quot;: 93,<br /> &quot;GroupsCount&quot;: 0,<br /> &quot;ReportedPeriods&quot;: {<br /> &quot;A&quot;: {<br /> &quot;Name&quot;: &quot;Annual&quot;,<br /> &quot;StartPeriod&quot;: &quot;1990&quot;,<br /> &quot;EndPeriod&quot;: &quot;2020&quot;<br /> }<br /> },<br /> &quot;Errors&quot;: true,<br /> &quot;ValidationReport&quot;: [<br /> {<br /> &quot;Type&quot;: &quot;Constraint&quot;,<br /> &quot;Errors&quot;: [<br /> {<br /> &quot;Message&quot;: &quot;Disallowed Dimension Value: REF_AREA=AFR&quot;,<br /> &quot;Dataset&quot;: 0,<br /> &quot;ComponentId&quot;: &quot; REF_AREA &quot;,<br /> &quot;ReportedValue&quot;: &quot;AFR&quot;,<br /> &quot;Position&quot;: &quot;Series&quot;,<br /> &quot;Keys&quot;: [&quot;AFR:BT:AA&quot;]<br /> }<br /> ]<br /> },<br /> {<br /> &quot;Type&quot;: &quot;Representation&quot;,<br /> &quot;Errors&quot;: [<br /> {<br /> &quot;Message&quot;: &quot;Dimension 'VARIABLE' is reporting value 'AA' which is not a valid representation in referenced Codelist 'OECD:CL_HIGH_AGLINK_2011_VARIABLE(1.0)'&quot;,<br /> &quot;Dataset&quot;: 0,<br /> &quot;Position&quot;: &quot;Series&quot;, <br /> &quot;ComponentId&quot;: &quot;VARIABLE&quot;,<br /> &quot;ReportedValue&quot;: &quot;AA&quot;,<br /> &quot;Keys&quot;: [&quot;AFR:BT:AA&quot;]<br /> },<br /> {<br /> &quot;Message&quot;: &quot;Error in Primary Measure 'OBS_VALUE': Reported value 'XXX' is not of expected type 'Double'&quot;,<br /> &quot;Dataset&quot;: 0, <br /> &quot;ComponentId&quot;: &quot; OBS_VALUE&quot;,<br /> &quot;ReportedValue&quot;: &quot;XXX&quot;,<br /> &quot;Position&quot;: &quot;Observation&quot;,<br /> &quot;Keys&quot;: [&quot;AFR:BT:IM:2010&quot;]<br /> }<br /> ]<br /> },<br /> {<br /> &quot;Type&quot;: &quot;FormatSpecific&quot;,<br /> &quot;Errors&quot;: [<br /> {<br /> &quot;Message&quot;: &quot;Unexpected attribute 'ASD' for element 'StructureSpecificData/DataSet/Series/Obs'&quot;,<br /> &quot;Dataset&quot;: 0,<br /> &quot;Position&quot;: &quot;Dataset&quot;<br /> }<br /> ]<br /> }<br /> ]<br /> }<br /> ],<br /> &quot;PreventsConversion&quot;: false,<br /> &quot;PreventsPublication&quot;: true<br /> }<br /> <br /> &lt;p&gt;Note the first three elements ‘Meta’, ‘InvalidData’, ‘ValidData’, there are present in the report if Inc-Metrics is set to true. Inc-valid and Inc-Invalid set to true enables the report to know the metrics for the invalid and valid data.&lt;/p&gt; <br /> &lt;p&gt;Note also each Error has a Type, this is the category of error which caused the validator to fail. For a list of all validators see the following section on Validators.<br /> The Error Position is either set to Dataset, Series, Observation, or Group. &lt;/p&gt;<br /> &lt;p&gt;PreventsConversion and PreventsPublication is an indication on the severity of the error. These settings on which errors prevent conversion and publication can be set in the Fusion Metadata Registry by the administrator of the system.&lt;/p&gt;<br /> <br /> == Dataset with an Unresolvable Datset Reference ==<br /> {<br /> &quot;FileFormat&quot;: &quot;Generic v2.1&quot;,<br /> &quot;MimeType&quot;: &quot;application/xml&quot;,<br /> &quot;Status&quot;: &quot;InvalidRef&quot;,<br /> &quot;Errors&quot;: true,<br /> &quot;Datasets&quot;: [{&quot;Dataflow&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:INVALID_DATAFLOW(1.0)&quot;}]<br /> }<br /> &lt;p&gt;&lt;b&gt;Errors&lt;/b&gt; has a value of true, the &lt;b&gt;Status&lt;/b&gt; states InvalidRef, and the &lt;b&gt;Datasets&lt;/b&gt; provides the reported reference which could not be resolved&lt;/p&gt;<br /> <br /> == Dataset which could not be read ==<br /> {<br /> &quot;Status&quot;: &quot;Error&quot;,<br /> &quot;Errors&quot;: true,<br /> &quot;Error&quot;: &quot;Unexpected '&lt;' character in element (missing closing '&gt;'?)\r\n at [row,col {unknown-source}]: [17,3]&quot;<br /> }<br /> &lt;p&gt;This error will be reported when the Fusion Metadata Registry is unable to determine what type of data the dataset is, so is unable to process the dataset for validation&lt;/p&gt;</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Data_Transformation_Web_Service&diff=6799 2022-12-21T12:48:18Z

<p>Mnelson: /* HTTP Headers */</p> <hr /> <div>[[Category:FMR REST API Reference]]<br /> = Overview =<br /> The Data Transformation Web Service converts the dataset submitted in the POST body to the data transmission format specified by the Accept header, optionally transforming it to a different Data Structure Definition if the Registry has a Structure Map defining the mapping.<br /> {| class=&quot;wikitable&quot;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Entry Point&lt;/b&gt;|| &lt;b&gt; /ws/public/data/transform&lt;/b&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Access&lt;/b&gt;|| &lt;span style='color:green'&gt;&lt;b&gt;Public&lt;/b&gt;&lt;/span&gt; (default). Configurable to Private<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Http Method&lt;/b&gt;|| POST<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accepts&lt;/b&gt;|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Compression&lt;/b&gt; || Zip files supported, if loading from URL gzip responses supported<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Content-Type&lt;/b&gt;|| &lt;p&gt;1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile&lt;/p&gt;<br /> &lt;p&gt;2. application/text or application/xml (if submitting data in the body of the POST)&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Format&lt;/b&gt;|| Determined by Accept Header - default SDMX 2.1 Structure Specific<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Statuses&lt;/b&gt;|| &lt;p&gt;&lt;b&gt;200&lt;/b&gt; - Transformation performed&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;400&lt;/b&gt; - Transformation could not be performed (either an unreadable datasets, or resolvable reference to a required structure)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;401&lt;/b&gt; - Unauthorized (if access has been restricted)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;500&lt;/b&gt; - Server Error&lt;/p&gt;<br /> |}<br /> <br /> = HTTP Headers =<br /> The Accept Header is used to define the output format, to transform the data to.<br /> <br /> In addition, the following optional header parameters can be used to provide further details on the incoming dataset. If these details are not provided, the Fusion Registry will interrogate the dataset header to get the information. If the dataset is a non-SDMX format, or does not contain the required information in the header, then an error response will be returned.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> !|HTTP Header || Purpose || Allowed Values<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accept&lt;/b&gt;|| The data transmission format to convert the dataset to.<br /> <br /> <br /> '''Note''': From FMR 11.5.0 the format (if not specified) defaults to the input format. Previous versions defaulted to SDMX Structure Specific 2.1<br /> ||<br /> &lt;p&gt;SDMX Formats&lt;/p&gt;<br /> &lt;ul dir=&quot;auto&quot;&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+csv;version=2.0.0;labels=[id|name|both];timeFormat=[original|normalized];keys=[none|obs|series|both]&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.genericdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.structurespecificdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.generictimeseriesdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.structurespecifictimeseriesdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+json;version=1.0.0&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+csv;version=1.0.0;labels=[id|both];timeFormat=[original|normalized]&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+edi&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;p&gt;SDMX Formats to be supported in future FMR releases&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+json;version=2.0.0&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+xml;version=3.0.0&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;p&gt;Note that the Fusion Excel data transmission format is supported as the input, but not output of a transformation.&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Data-Format&lt;/b&gt;|| Used to inform the server when the data is in CSV format. ||csv;delimiter=[delimiter]<br /> &lt;p&gt;Where [delimiter] is either:&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;comma&lt;/li&gt;<br /> &lt;li&gt;tab&lt;/li&gt;<br /> &lt;li&gt;semicolon&lt;/li&gt;<br /> &lt;li&gt;space&lt;/li&gt;<br /> &lt;/ul&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Structure&lt;/b&gt; || &lt;p&gt;(optional) Provides the structure to validate the data against.&lt;/p&gt;<br /> <br /> &lt;p&gt;This is optional as this information may be present in the header of the DataSet. If provided this value will override the value in the dataset (if present).&lt;/p&gt;<br /> || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Receiver-Id&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||<br /> &lt;p&gt;The ReceiverId may be included in the validation report.&lt;/p&gt;<br /> &lt;p&gt;If not provided, the ReceiverId will be taken from the header of the dataset if it is present. &lt;/p&gt;<br /> &lt;p&gt;If the dataset does not contain a ReceiverId (for example a non-SDMX format) then the validation report will not contain a ReceiverId in the header.&lt;/p&gt;<br /> ||<br /> The following characters are allowed: A-z, a-z 0-9 $, _, -, @, \<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Structure&lt;/b&gt;||&lt;p&gt;Provides the structure used to read the data.&lt;/p&gt;<br /> &lt;p&gt;This is optional as this information may be present in the header of the DataSet. If provided this value will override the value in the dataset (if present).&lt;/p&gt; <br /> ||Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition.<br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Idx&lt;/b&gt;||&lt;p&gt;If the loaded file contains multiple datasets, this argument can be used to indicate which dataset is transformed. If this argument is not present then all datasets will be in the output file (if the file formats permits multiple datasets).&lt;/p&gt;<br /> ||Zero indexed integer, example: 0<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Id&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;An optional parameter which allows the user to specify the value of the DataSetID generated in the validation.<br /> ||<br /> The following characters are allowed:<br /> A-z, a-z<br /> 0-9<br /> $, _, -, @, \<br /> Specific variables permit the insertion of Data Structure / Data Flow values. These values are:<br /> &lt;br/&gt;${DATFLOW_ID}<br /> &lt;br/&gt;${DATFLOW_ACY}<br /> &lt;br/&gt;${DATFLOW_VER}<br /> &lt;br/&gt;${DSD_ID}<br /> &lt;br/&gt;${DSD_ACY}<br /> &lt;br/&gt;${DSD_VER}<br /> &lt;p/&gt;<br /> Note that dots in the version number will be replaced with the _ character, since dots are not permitted in the ID.&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Action&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8.1)&lt;/p&gt;<br /> ||An optional parameter which allows the user to specify the value of the DataSetAction generated in the validation report. If this parameter is not specified, the default value will be used.<br /> ||May be one of the following: <br /> * Append <br /> * Replace <br /> * Merge <br /> * FullReplace <br /> * Delete <br /> * Information<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Map-Structure&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.2.13)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;An optional parameter to inform the Fusion Registry to transform the structure of the dataset to conform to another Data Structure Definition.&lt;/p&gt;<br /> &lt;p&gt;The value provided can be a URN of a Dataflow or Data Structure Definition to map the incoming data to. A Structure Map must exist in the Fusion Registry which maps between the incoming Data Structure/Dataflow and Mapped Data Structure/Dataflow.&lt;/p&gt;<br /> &lt;p&gt;Alternatively the URN may be the URN of the Data Structure Map to use for the mapping (since v9.4.4)&lt;/p&gt;<br /> ||<br /> Valid SDMX URN for Dataflow or Data Structure Definition.<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Unmapped&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;If the Map-Structure Header is used, then the inclusion of Inc-Unmapped will output a second dataset, if there are unmapped series. The additional dataset contains the data that could not be mapped due to missing mapping rules, or ambiguous outputs.&lt;/p&gt;<br /> &lt;p&gt;The format of the additional dataset is the same format as the output dataset.&lt;/p&gt;<br /> &lt;p&gt;As the result may contain a separate file, the response format is either set to multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file. The file names are 'out' and 'unmapped' with the file extension based on the output format.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-UnmappedReport&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v11.5.0)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;If the Map-Structure Header is used, then the inclusion of Inc-UnmappedReport may output another file, if there are unmapped series. The additional file contains a report on the information that could not be mapped due to missing mapping rules, or ambiguous outputs.&lt;/p&gt;<br /> &lt;p&gt;The format of this report consists of JSON elements:&lt;/p&gt;<br /> * The StructureMap used in the mapping<br /> * The Source Structure URN<br /> * The Target Structure URN<br /> * The Result<br /> &lt;p&gt;<br /> The result consists of an Input and an Output which details what the input managed to map to. The output also contains a Array called &quot;MissingDimensions&quot; which lists the ID of the missing dimensions.<br /> ||Boolean (true/false)<br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Metrics&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Includes metrics on the transformation.&lt;/p&gt;<br /> &lt;p&gt;The result will contain a separate file, either as a multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> <br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Fail-On-Error&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.5.0)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;An optional parameter to tell the transformation process to fail if an error is detected in the dataset.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Zip&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Compresses the output as a zip file. This if used in conjunction with Inc-Metrics or Inc-Unmapped the zip will contain multiple files.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Duplicate-Behaviour&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v11.1.6)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Specify the behaviour to perform when duplicate observations are encountered. Either the duplicates can be preserved or either the first or last value can be used.&lt;/p&gt;<br /> ||May be one of the following: <br /> * useFirst <br /> * useLast<br /> * preserve<br /> |}<br /> <br /> <br /> == Include Metrics ==<br /> The following JSON is an example response when Inc-Metrics header is set to true. Request Time is Epoc Time Milliseconds, and Duration is measured in the number of milliseconds taken to complete the transformation.<br /> <br /> { <br /> &quot;Meta&quot;: { <br /> &quot;RequestTime&quot;: 1559124708568, <br /> &quot;Duration&quot;: 220 }, <br /> &quot;SourceData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { <br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:IN_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 3118, <br /> &quot;Observations&quot;: 3118, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> }, <br /> &quot;OutputData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { <br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:OUT_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 1753, <br /> &quot;Observations&quot;: 1855, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> }, <br /> &quot;UnMappedData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:IN_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 1263, <br /> &quot;Observations&quot;: 1263, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> } <br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=Data_Transformation_Web_Service&diff=6798 2022-12-21T12:47:07Z

<p>Mnelson: /* HTTP Headers */</p> <hr /> <div>[[Category:FMR REST API Reference]]<br /> = Overview =<br /> The Data Transformation Web Service converts the dataset submitted in the POST body to the data transmission format specified by the Accept header, optionally transforming it to a different Data Structure Definition if the Registry has a Structure Map defining the mapping.<br /> {| class=&quot;wikitable&quot;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Entry Point&lt;/b&gt;|| &lt;b&gt; /ws/public/data/transform&lt;/b&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Access&lt;/b&gt;|| &lt;span style='color:green'&gt;&lt;b&gt;Public&lt;/b&gt;&lt;/span&gt; (default). Configurable to Private<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Http Method&lt;/b&gt;|| POST<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accepts&lt;/b&gt;|| CSV, XLSX, SDMX-ML, SDMX-EDI (any format for which there is a Data Reader)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Compression&lt;/b&gt; || Zip files supported, if loading from URL gzip responses supported<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Content-Type&lt;/b&gt;|| &lt;p&gt;1. multipart/form-data (if attaching file) – the attached file must be in field name of uploadFile&lt;/p&gt;<br /> &lt;p&gt;2. application/text or application/xml (if submitting data in the body of the POST)&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Format&lt;/b&gt;|| Determined by Accept Header - default SDMX 2.1 Structure Specific<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Response Statuses&lt;/b&gt;|| &lt;p&gt;&lt;b&gt;200&lt;/b&gt; - Transformation performed&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;400&lt;/b&gt; - Transformation could not be performed (either an unreadable datasets, or resolvable reference to a required structure)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;401&lt;/b&gt; - Unauthorized (if access has been restricted)&lt;/p&gt;<br /> &lt;p&gt;&lt;b&gt;500&lt;/b&gt; - Server Error&lt;/p&gt;<br /> |}<br /> <br /> = HTTP Headers =<br /> The Accept Header is used to define the output format, to transform the data to.<br /> <br /> In addition, the following optional header parameters can be used to provide further details on the incoming dataset. If these details are not provided, the Fusion Registry will interrogate the dataset header to get the information. If the dataset is a non-SDMX format, or does not contain the required information in the header, then an error response will be returned.<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> !|HTTP Header || Purpose || Allowed Values<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Accept&lt;/b&gt;|| The data transmission format to convert the dataset to.<br /> <br /> <br /> '''Note''': From Fusion Registry 11.8.0 the format (if not specified) defaults to the input format. Previous versions defaulted to SDMX Structure Specific 2.1<br /> ||<br /> &lt;p&gt;SDMX Formats&lt;/p&gt;<br /> &lt;ul dir=&quot;auto&quot;&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+csv;version=2.0.0;labels=[id|name|both];timeFormat=[original|normalized];keys=[none|obs|series|both]&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.genericdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.structurespecificdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.generictimeseriesdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.structurespecifictimeseriesdata+xml;version=2.1&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+json;version=1.0.0&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+csv;version=1.0.0;labels=[id|both];timeFormat=[original|normalized]&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+edi&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;p&gt;SDMX Formats to be supported in future FMR releases&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+json;version=2.0.0&lt;/li&gt;<br /> &lt;li&gt;application/vnd.sdmx.data+xml;version=3.0.0&lt;/li&gt;<br /> &lt;/ul&gt;<br /> &lt;p&gt;Note that the Fusion Excel data transmission format is supported as the input, but not output of a transformation.&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Data-Format&lt;/b&gt;|| Used to inform the server when the data is in CSV format. ||csv;delimiter=[delimiter]<br /> &lt;p&gt;Where [delimiter] is either:&lt;/p&gt;<br /> &lt;ul&gt;<br /> &lt;li&gt;comma&lt;/li&gt;<br /> &lt;li&gt;tab&lt;/li&gt;<br /> &lt;li&gt;semicolon&lt;/li&gt;<br /> &lt;li&gt;space&lt;/li&gt;<br /> &lt;/ul&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Structure&lt;/b&gt; || &lt;p&gt;(optional) Provides the structure to validate the data against.&lt;/p&gt;<br /> <br /> &lt;p&gt;This is optional as this information may be present in the header of the DataSet. If provided this value will override the value in the dataset (if present).&lt;/p&gt;<br /> || Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Receiver-Id&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||<br /> &lt;p&gt;The ReceiverId may be included in the validation report.&lt;/p&gt;<br /> &lt;p&gt;If not provided, the ReceiverId will be taken from the header of the dataset if it is present. &lt;/p&gt;<br /> &lt;p&gt;If the dataset does not contain a ReceiverId (for example a non-SDMX format) then the validation report will not contain a ReceiverId in the header.&lt;/p&gt;<br /> ||<br /> The following characters are allowed: A-z, a-z 0-9 $, _, -, @, \<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Structure&lt;/b&gt;||&lt;p&gt;Provides the structure used to read the data.&lt;/p&gt;<br /> &lt;p&gt;This is optional as this information may be present in the header of the DataSet. If provided this value will override the value in the dataset (if present).&lt;/p&gt; <br /> ||Valid SDMX URN for Provision Agreement, Dataflow, or Data Structure Definition.<br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Idx&lt;/b&gt;||&lt;p&gt;If the loaded file contains multiple datasets, this argument can be used to indicate which dataset is transformed. If this argument is not present then all datasets will be in the output file (if the file formats permits multiple datasets).&lt;/p&gt;<br /> ||Zero indexed integer, example: 0<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Id&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8)&lt;/p&gt;||&lt;p&gt;An optional parameter which allows the user to specify the value of the DataSetID generated in the validation.<br /> ||<br /> The following characters are allowed:<br /> A-z, a-z<br /> 0-9<br /> $, _, -, @, \<br /> Specific variables permit the insertion of Data Structure / Data Flow values. These values are:<br /> &lt;br/&gt;${DATFLOW_ID}<br /> &lt;br/&gt;${DATFLOW_ACY}<br /> &lt;br/&gt;${DATFLOW_VER}<br /> &lt;br/&gt;${DSD_ID}<br /> &lt;br/&gt;${DSD_ACY}<br /> &lt;br/&gt;${DSD_VER}<br /> &lt;p/&gt;<br /> Note that dots in the version number will be replaced with the _ character, since dots are not permitted in the ID.&lt;/p&gt;<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Dataset-Action&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.8.1)&lt;/p&gt;<br /> ||An optional parameter which allows the user to specify the value of the DataSetAction generated in the validation report. If this parameter is not specified, the default value will be used.<br /> ||May be one of the following: <br /> * Append <br /> * Replace <br /> * Merge <br /> * FullReplace <br /> * Delete <br /> * Information<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Map-Structure&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.2.13)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;An optional parameter to inform the Fusion Registry to transform the structure of the dataset to conform to another Data Structure Definition.&lt;/p&gt;<br /> &lt;p&gt;The value provided can be a URN of a Dataflow or Data Structure Definition to map the incoming data to. A Structure Map must exist in the Fusion Registry which maps between the incoming Data Structure/Dataflow and Mapped Data Structure/Dataflow.&lt;/p&gt;<br /> &lt;p&gt;Alternatively the URN may be the URN of the Data Structure Map to use for the mapping (since v9.4.4)&lt;/p&gt;<br /> ||<br /> Valid SDMX URN for Dataflow or Data Structure Definition.<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Unmapped&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;If the Map-Structure Header is used, then the inclusion of Inc-Unmapped will output a second dataset, if there are unmapped series. The additional dataset contains the data that could not be mapped due to missing mapping rules, or ambiguous outputs.&lt;/p&gt;<br /> &lt;p&gt;The format of the additional dataset is the same format as the output dataset.&lt;/p&gt;<br /> &lt;p&gt;As the result may contain a separate file, the response format is either set to multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file. The file names are 'out' and 'unmapped' with the file extension based on the output format.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-UnmappedReport&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v11.5.0)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;If the Map-Structure Header is used, then the inclusion of Inc-UnmappedReport may output another file, if there are unmapped series. The additional file contains a report on the information that could not be mapped due to missing mapping rules, or ambiguous outputs.&lt;/p&gt;<br /> &lt;p&gt;The format of this report consists of JSON elements:&lt;/p&gt;<br /> * The StructureMap used in the mapping<br /> * The Source Structure URN<br /> * The Target Structure URN<br /> * The Result<br /> &lt;p&gt;<br /> The result consists of an Input and an Output which details what the input managed to map to. The output also contains a Array called &quot;MissingDimensions&quot; which lists the ID of the missing dimensions.<br /> ||Boolean (true/false)<br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Inc-Metrics&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Includes metrics on the transformation.&lt;/p&gt;<br /> &lt;p&gt;The result will contain a separate file, either as a multipart/mixed message with a boundary per file, or if the Zip header is set to true, the output will be a single zip file.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> <br /> <br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Fail-On-Error&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.5.0)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;An optional parameter to tell the transformation process to fail if an error is detected in the dataset.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Zip&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v9.6.5)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Compresses the output as a zip file. This if used in conjunction with Inc-Metrics or Inc-Unmapped the zip will contain multiple files.&lt;/p&gt;<br /> ||Boolean (true/false)<br /> |-<br /> |style=&quot;background-color:#eaecf0&quot;|&lt;b&gt;Duplicate-Behaviour&lt;/b&gt;&lt;p style=&quot;font-size:0.85em&quot;&gt;(Since v11.1.6)&lt;/p&gt;<br /> ||<br /> &lt;p&gt;Specify the behaviour to perform when duplicate observations are encountered. Either the duplicates can be preserved or either the first or last value can be used.&lt;/p&gt;<br /> ||May be one of the following: <br /> * useFirst <br /> * useLast<br /> * preserve<br /> |}<br /> <br /> <br /> == Include Metrics ==<br /> The following JSON is an example response when Inc-Metrics header is set to true. Request Time is Epoc Time Milliseconds, and Duration is measured in the number of milliseconds taken to complete the transformation.<br /> <br /> { <br /> &quot;Meta&quot;: { <br /> &quot;RequestTime&quot;: 1559124708568, <br /> &quot;Duration&quot;: 220 }, <br /> &quot;SourceData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { <br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:IN_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 3118, <br /> &quot;Observations&quot;: 3118, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> }, <br /> &quot;OutputData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { <br /> &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:OUT_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 1753, <br /> &quot;Observations&quot;: 1855, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> }, <br /> &quot;UnMappedData&quot;: { <br /> &quot;Datasets&quot;: [<br /> { &quot;Structure&quot;: &quot;urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=BIS:IN_FLOW(1.0)&quot;, <br /> &quot;Series&quot;: 1263, <br /> &quot;Observations&quot;: 1263, <br /> &quot;Groups&quot;: 0 <br /> }<br /> ] <br /> } <br /> }</div>

Mnelson https://fmrwiki.sdmxcloud.org/index.php?title=FMR_Workbench&diff=6796 2022-12-09T10:37:51Z

<p>Mnelson: /* Overview */</p> <hr /> <div>= Overview =<br /> [[File:Fmr wb overview.png|thumb]]<br /> The FMR Workbench is a web application which provides the web user interface to view and maintain structural metadata, the critical difference between the FMR Workbench and the Fusion Metadata Registry is that the FMR Workbench does not have a database - the structures that it views and maintains come from a separate web service.<br /> <br /> The FMR Workbench reads a config file on startup which describes one or more 'services'. Each service is an SDMX endpoint with a query API and an optional persistence API. The web user interface of the FMR Workbench enables users to switch between services. The user can then browse the SDMX Structures of the service they are connected to.<br /> <br /> The connected service is attached to the user's session, therefore it is possible for multiple users to use the same FMR-Workbench, with each user connecting to a different service.<br /> <br /> The FMR-Workbench can be used to edit, and save structures back to the service, if the service supports persistence, and the user is logged in.<br /> <br /> = Switching Services =<br /> [[File:Fmr wb service.png|thumb]]<br /> The header bar of the FMR-Workbench contains a button showing the currently connected service, clicking on this button will open a window to allow a different service to be connected. By default the service is connected to anonymously, meaning only browser functionality will be supported.<br /> <br /> To authenticate, click on the Login button after connecting to the service. <br /> <br /> = Authentication =<br /> The FMR Workbench supports 2 forms of authentication with the connected service: Basic Authentication and OpenID Connect. The type of authentication is set up in the Config file of the FMR-Workbench. <br /> <br /> == Basic Authentication ==<br /> Clicking Login will take the user to the FMR-Workbench login page where the user is invited to enter a username and password. These details are stored in the FMR-Workbench against the user's session, and they are used whenever the user interacts with the target service. An initial check is performed against the target server to ensure the credentials can be used to obtain some structures, if the check fails then the details are deemed to be invalid and will be discarded.<br /> <br /> == Open ID Connect ==<br /> Clicking Login will take the user to the OpenID Connect login page (defined in the configuration file of the FMR-Workbench). The user authenticates with the chosen OpenID Connect authentication provider, on success the user is redirected back to the FMR-Workbench with a token. The token is stored against the user's session, and used to authenticate with the target service. <br /> <br /> = Structure Navigation and Export =<br /> The FMR-Workbench front end will display structures in the same way the FMR provides - clicking on a sidebar item (example Codelists) will display the Codelist page and all the Codelists in the connected service. This is achieved by the front end issuing a SDMX web service query to the FMR Workbench for all Codelists as stubs. This query is then forwarded to the connected service, and the response is written back to the client. The FMR-Workbench can receive the query response from the connected service in any SDMX format. The response will be parsed by the FMR-Workbench before being written out in the requested format. This means the FMR-Workbench is able to write the response out in ANY format that the FMR Workbench supports, including Excel, SDMX, and JSON. The user is therefore able to click on a Codelist and export it as Excel, even if the connected service does not support Excel as an output format.<br /> <br /> = SDMX Web Service =<br /> It is important to note that the FMR-Workbench has the same web service entry points as the FMR, and whilst these web services work, they are not guaranteed to be fully compliant. The web services of the FMR-Workbench forwards queries onto the connected service. The queries may be modify to reduce the possibility of the target service not supporting the query. If the connected service reports any error in the query, the FMR-Workbench will report 'no structures' for the query. <br /> <br /> The recommendation is therefore to not rely on the web services of the FMR-Workbench, instead use the web services provided by the target service directly. <br /> <br /> = Structure Creation/Modification =<br /> If the user has logged into the connected service, they will be able to upload structures from the home page of the FMR-Workbench, or create/edit structures using the standard Wizards in the User Interface. <br /> <br /> It is important to note that the FMR-Workbench contains the same User Interface as the FMR, and as such does not provision for the fact that the connected service may not support all the features of the FMR. For example it may be possible in the FMR-Workbench to create a SDMX 3.0 DSD with multiple measures, but the target service may reject the save request.<br /> <br /> It is possible to load structures in any format supported by the FMR-Workbench, including Excel - these structures will be converted into a format supported by the connected service on the save request.</div>

Mnelson