Change Log

Changelog for Fusion Metadata Registry

'''Currently working on this - val - 22.01.2021'''


Version 10.6.3  (Released 2021/01/22)
Change:
[FR-4904] - Change Kafka Service to allow status to be Publicly Available

Bug Fixes:
[FR-4907] - Saving a DSD with a non-standard Measure Dimension prevents the DSD from being subsequently edited
[FR-4906] - Requesting an SDMX file which cannot be described in SDMX throws an error
Bug Fixes:
[FR-4893] - Activity -> Structure Submissions page show a spinner if item selected in table and user not logged in

Bug Fixes:
[Fr-4891] - NPE on startup when extended Codelist no longer exists
Bug Fixes:
[FR-4884] - DSD Wizard issues with re-assigning Time Dimension
New Features:
[FR-4863] - Kafka - add email user on forced producer termination \(when fatal error occurs\) 
[FR-4861] - Kafka - support filter structures from notification by structure type 
[FR-4860] - Kafka - record notificaiton status against registry transaction status. Restart structure producer from last tx.
[FR-4859] - Kafka - support all connection properties when defining kafka server connection

Improvements:
[FR-4865] - Kaka - improve the test connection feature
[FR-4864] - Kafka - Terminate structure producer on fatel error
[FR-4862] - Kafka - Do not flush all structures on startup or kafka configuration change

Changes:
[FR-4877] - Tableau - split time period column into 2 columns \(start period and end period\)

Bug Fixes:
[FR-4879] - Schema Generation: Group Type does not match standards
[FR-4867] - UI delete and add TIME_PERIOD Concept with different Id causes Dimension to be misclassified 
[FR-4866] - DSD Measure can be lost when updating structure
[FR-4840] - Loading an EDI file which has invalid values returns a 500 rather than throwing a FormatSpecific Error
[FR-4837] - Ensure CSV Dump honours double quotes around values containing commas
[FR-4830] - Outputting JSON data with Annotations can result in a -1 being returned
Bug Fixes:
[FR-4787] - EDI File containing no department ID in CTA block cannot be validated
[FR-4788] - Dataset ID is not copied to outputted SDMX JSON
[FR-4789] - Activity -> Structure Submission Page - Compare button: nothing happens when clicked
[FR-4790] - If a user with no permissions is created, attempting to log on results in a Stack Trace on the screen
[FR-4792] - Unable to create a Dynamic DSD Attribute with multiple component inputs
[FR-4793] - EDI files are being created with message identity blocks greater than 35 characters and these are not erroring when read back in
Improvement:
[FR-4780] - The Registry can unify monthly data so reading and writing comes out in a consistent fashion

Bug Fix:
[FR-4771] - Temporary file not cleared away after reading Excel file


-----------------

Version 10.5.7  (Released 2020/11/13)
New Features:
[FR-4768] - REST - normalise frequency support first in period and last in period functions
[FR-4753] - Registry now has option to clean up temporary directory on startup
[FR-4712] - Format of Code IDs can be restricted for a Codelist
[FR-4689] - Data Validation - support multiple structure references on header to inditify which Dataflows or Provisions to use in validation when multiple datasets are in file
[FR-4673] - New parameter on Data Conversion to Merge Datasets
[FR-4666] - Add SSL support to Kafka connection 
[FR-4670] - Add a System Property to ignore EDI Agency checks
[FR-4623] - Reporting Template: Hierarchy in row, codes with no data which have child codes with data to be displayed as a presentation node
[FR-4640] - Reporting Template Wrap text if it exceeds 'n' characters
[FR-4617] - Excel Reporting Template: Support Instruction sheets
[FR-4653] - Create property allowing DSI field in EDI file to be resolved to a Dataflow
[FR-4616] - Excel Reporting Template: Display of Codes and Description in the same field
[FR-4619] - Excel Reporting Templates: Protect the workbook so users cannot change the name of the worksheets or add/delete new worksheets
[FR-4597] - Web Service for Modifying and Obtaining System Properties
[FR-4477] - Allow HTTP Read Timeout duration to be viewed and modified via UI  
[FR-4478] - Allow HTTP Connect Timeout duration to be viewed and modified via UI 
[FR-4460] - View and modify User-Agent used in HTTP Requests from the Registry via the User Interface  
[FR-4378] - REST API - Support path parameter: allowedconstraint
[FR-4379] - REST API - Support path parameter: allowedconstraint
[FR-4380] - REST API - Support query parameter: allcompletestubs
[FR-4381] - REST API - Support query parameter: referencecompletestubs       
[FR-4394] - Exporting a DSD to EDI - support feature to enforce Primary Measure [https://wiki.sdmxcloud.org/Edi_Enforce_Primary_Measure]
[FR-4409] - EDI Lenient mode - allow DSDs to be read and written with uncoded dimensions [https://wiki.sdmxcloud.org/Edi_Lenient_Mode]
[FR-4408] - Add Value list to Bulk Export
[FR-4369] - Support Constraints against Value List
[FR-3302] - DSD Changes: Support zero to many measures - as per SDMX 3.0 working specification
[FR-4276] - Bulk Download offers SDMX-JSON in addition to Fusion-JSON as a format
[FR-4336] - Support Enumeration of items which can have non-SDMX valid IDs (Value List)
[FR-4337] - Reverse Engineer DSD from CSV Dataset
[FR-3305] - Populate an Excel Reporting Template from an SDMX Data File
[FR-2879] - full support for the official SDMX-JSON 1.0 standard 
[FR-4114] - AD role mappings CSV import and export
[FR-3304] - Codelist inheritance
[FR-4115] - External table name mapping
[FR-4029] - CORS policy improvements
[FR-3946] - Stronger AES256 encryption 
[FR-3561] - Kafka integration – Fusion Registry producer for structural metadata (including Kerberos integration for security)  
[FR-4159] - Change the ID and Agency of existing structures 
[FR-4072] - Secondary level caching for External SQL datastores, and mapped Dataflows
[FR-4099] - Environment Synchronisation accepts user log in allowing it to work with private or content-restricted registries
[FR-3967] - Active Directory configuration custom search filter
[FR-3951] - New Data Validator: Structure Validator
[FR-3962] - New Data Validator: Obs Status Validation
[FR-3892] - "Missing Time Periods" is its own validator 

Improvement:
[FR-4770] - Add support to Fusion CSV to load a CSV file declaring Dimensions and measure dimensions only
[FR-4755] - New Web Service to allow users to 'Close' loaded Data
[FR-4732] - Adding Error Codes to Fusion Registry Validation Response
[FR-4748] - Support Loading of Fusion CSV Data
[FR-4750] - Report errors in EDI attribute validation more clearly
[FR-4730] - Enforce that the EDI FNS block is followed by the correct value
[FR-4688] - Remove 'zombie structures' from registry content, enable users to export or delete via a separate API ??????
[FR-4303] - Kafka Connection Configuration - support multiple hosts
[FR-4660] - Support Quarterly and Half-Yearly formats in SDMX 1.0
[FR-4661] - Upgrade Excel to Structure Map creator to support Fusion Data Mapper v1.1.0
[FR-4652] - Report Templates: Conditional Colour on Attributes - make optional (opt in)
[FR-4632] - Improve Export Structures modal to warn if structures will not be exported in chosen format
[FR-4639] - Enhancement to calculation and interpretation of EDI DSI segment
[FR-4549] - Add 'Reference Partial' option to the export modal on bulk actions
[FR-4615] - Excel Reporting Template: Do not use color on attributes section
[FR-4581] - Add support for Groups in SDMX CSV
[FR-4587] - SDMX JSON Does not support the writing of Groups
[FR-4588] - SDMX JSON Does not support the reading of Groups
[FR-4590] - Issues with Monthly format and SDMX 2.0
[FR-4579] - Enhance the RSS feed to list URLs of added, modified and deleted structures
[FR-4580] - Improved performance in EDI Data Reader
[FR-4511] - Change VND Header for EDI to be consistent with SDMX structure and data syntax
[FR-4512] - Order Excel Report Template Colors in Legend by Codelist Code order
[FR-4214] - Add 'Export SDMX-ML' button to structure pages 
[FR-4411] - Improve performance of cross reference resolution
[FR-4363] - Throw HTTP 406 (Not Acceptable) if a structure query explicitly requests the response in a format which is understood but not supported by any structures in the output
[FR-4361] - Simplify install wizard
[FR-4324] - Performance Improvements for Data Convert
[FR-4331] - Show the Hierarchical Codelist Agency/ID/Version of available Hierarchies in Validation Rules Wizard
[FR-4321] - EDI to treat DSI as Dataflow Identifier on Data Read
[FR-4322] - Report Template vnd header - if Data Provider Id is supplied and Ageny is not, default Agency to SDMX
[FR-4329] - Remove formula embedded in Report Templates (replaced with checking table)
[FR-4263] - Reporting Template Worksheets to be built from a Dataflow as oppose to a Content Constraint
[FR-4227] - Sdmx JSON meta tag needs to have required node of "sender"
[FR-3882] - Registry should allow Half-Yearly to be specified as code "H"

Removed Feature:
[FR-4081] - Remove: In-memory structures storage

Bug Fixes:
[FR-4759] - Header ID can be incorrectly longer than 10 characters
[FR-4763] - CTA Segment should only be output in EDI if it is valid
[FR-4731] - InputStreams being incorrectly left open
[FR-4733] - Temporary files being created and not cleared away
[FR-4749] - POST requests ending with a slash prevent POST operation from completing successfully
[FR-4727] - Ensure logout after install of Fusion Registry
[FR-4707] - Structure Map - Value Map with Regular expression rejected as expression exceeds max length of Component
[FR-4709] - Unify Rest API VND Headers
[FR-4718] - Issue with format conversion to EDI when an observation level attribute is the only content of an observation
[FR-4701] - Codelist Wizard: re-ordering codes loses non-English languages for those codes
[FR-4702] - Codelist Wizard Step 4, clicking outside of the pop-up window dismisses window and reverts user changes
[FR-4703] - Saving a CSV Locale with 'no value' for the Group Separator value fails
[FR-4704] - StaxReader keeping InputStream open preventing temp file deletion
[FR-4705] - Codelist Wizard Step 4 - Show CSV does not honour languages correctly
[FR-4706] - Concept Scheme Wizard Step 4 - Show CSV does not honour languages correctly
[FR-4698] - Data Provider Constraint linked to Dimension which no longer exists causes registry startup to fail
[FR-4685] - EDI - observations not included in output when a series is written which has a start date earlier then any previous series of the same freequency, AND an end date later then any series of the same frequency
[FR-4674] - SDMX 2.0 data incorrectly stating KeyFamilyURI is an invalid attribute
[FR-4664] - Null pointer when saving a DSD with only 1 measure which does not have Id of OBS_VALUE
[FR-4658] - EDI DSI block should not be permitted to be longer than 18 characters
[FR-4655] - Install page of Fusion Registry - entering an invalid server URL completes the installation
[FR-4656] - Unable to specify space as locale Group Separator on CSV Locale page
[FR-4523] - Updating a DSD with optional Observation Attribute causes Index Out Of Bounds error if data already exists
[FR-4624] - IndexOutOfBounds can be thrown by validating data file
[FR-4626] - SDMXJsonDataReaderEngine is not correctly parsing observations when data file has groups
[FR-4631] - Changing the ID of a structure does not allow it to be obtained by ETag
[FR-4633] - Reporting Template - universe of series miscalculation when including a excluded series constraint with wildcards
[FR-4605] - CSV format (series per column) does not support hebrew characters
[FR-4607] - SDMX JSON - Unknown Attributes at the Series Level should throw an error
[FR-4612] - Half-Yearly issue error in Data handling
[FR-4613] - Issues with Monthly format and SDMX 2.1
[FR-4554] - When editing DSD, unable to change OBS_VALUE to another agency
[FR-4584] - EDI issue - FNS block must be followed by a REL
[FR-4596] - Recursive loop on reindex when 2 provisions linked to the same mapped dataflow
[FR-4598] - Ensure SDMX JSON and SDMX CSV do not support the 1 digit week format
[FR-4599] - Creating Schema can throw Null Pointer Exception
[FR-4524] - Weekly format incorrect for single digit weeks
[FR-4526] - Transforming data files with multiple Datasets should set the header action correctly
[FR-4527] - Converting file should create Group Type of SiblingType
[FR-4535] - Converting file to EDI loses Weekly Data
[FR-4537] - Ensure Generic Data Readers are version independant
[FR-4545] - Support SDMX Dates for Monthly format with the "M" character
[FR-4517] - Empty IDE block following NAD+MS should not be treated as an error
[FR-3281] - Organisations tab - Data Providers not properly sorted
[FR-4495] - JSON data file returning Format Specific error rather than Representation Error
[FR-4496] - Format Specific Errors should not all be at Dataset Level
[FR-4497] - EDI data file returning Format Specific Error rather than Representation error
[FR-4508] - Validation Service should return a 400 when supplied DSD is unknown
[FR-4510] - Role Mappings Organisations should be sorted alphabetically
[FR-4513] - Memory leak when generating Reporting Templates
[FR-4515] - Series Constraint Wizard - step 2 - requires the ability to select multiple Provision Agreements
[FR-4495] - JSON data file returning Format Specific error rather than Representation Error
[FR-4501] - gracefully handle startup failures due to missing internal references
[FR-4474] - Editing a DSD can cause Series and Group information to be lost
[FR-4479] - Hierarchical Codelist fusion-json level incorrectly outputs number, not the level Id that is referenced
[FR-4401] - Validation service: Duplicate attribute values
[FR-4458] - FormatSpecificError expected when reading CSV file
[FR-4461] - JSON: Unknown Attribute at the Observation level should throw Structure Error
[FR-4400] - EDI: Fix to reading of characters which fall outside of the Ascii range
[FR-4459] - EDI: TimePeriodFormat exception should be thrown when analysing attributes
[FR-4462] - EDI: Extra FNS segment causes issues
[FR-4463] - EDI: No IDE segment following the ARR one
[FR-4464] - EDI: Data segment allowed in Attributes section
[FR-4465] - SDMX-CSV data loaded via User Interface as Data Format: CSV with No DSD selected can cause Dataflow lookup error when enclosed in double quotes
[FR-4466] - Unable to load SDMX-CSV via UI when explicit DSD is selected
[FR-4467] - Data files with Weekly frequency can fail to be processed by the Registry
[FR-4471] - CSV - format conversion service is failing due to BOM issue
[FR-4470] - JSON: Format conversion issue - dataset attributes are dropped when converting to JSON data format
[FR-4475] - JSON: Representation error expected due to facet violation but no error thrown
[FR-4480] - SDMX-JSON Hierarchical Codelist Level reference should be by URN - currently it is by Id
[FR-4483] - User unable to create DSD when using "Reverse Engineer DSD from CSV Dataset" due to unclear Error
[FR-4487] - Bug in legacy mode (sdmx-json=fusion-json) for Reference Metadata
[FR-4488] - Bug in legacy mode for availability API
[FR-4482] - REST API references=all is returning descendants of parent structures, should only be children
[FR-4489] - REST API -with Fusion Store: 'offset' parameter used in conjuntion with 'max' parmaeter, is not honoured when 'lastNObservations' is included in query
[FR-4268] - Converting Data - timeFormat = "normalized" - should always return time in ISO 8601
[FR-4292] - HCL Wizard formal Levels, resave causes error in level ID
[FR-4391] - Default value for version not respected in SDMX 2.1 dataset - structure reference is changed to latest version (should be 1.0)
[FR-4392] - IDE segment should only be used to look up DataStructrure (previously defaulted to Dataflow if a match was found)
[FR-4393] - Regression - Data validation for Representation missing from on Group attributes
[FR-4395] - Generic 2.0 files must permit "DataSet" attribute to be in namespace "Generic"
[FR-4399] - EDI: Uncoded Attribute Values must be checked to ensure they are valid EDI characters
[FR-4407] - WS Structure Query - Value List is not on dropdown
[FR-4412] - REST API Delete Method does not support non-SDMX structures
[FR-4420] - Value List - Export to Excel (from VL Page) not working
[FR-4422] - Value List - Annotations - Post add - Page not found
[FR-4423] - View Provision Agreement - Spinner does not disappear (intermittent)
[FR-4441] - referencepartial to treat constraint references as a union, not intersection
[FR-4449] - EDI Delete messages should throw an error if they contain observations or observation attributes
[FR-4453] - Constraint Wizard - edit constraint to add new attachment and remove another results in server side error
[FR-4454] - Illegal Value in EDI contact not reported correctly
[FR-4455] - Supplying an invalid DataStructure in EDI header causes a NullPointerException
[FR-4456] - converting a version 1.0 dataset to a version 2.0 keeps the action as Update (should convert to replace)
[FR-4373] - EDI CTA segment should throw error if Contact Name contains invalid characters
[FR-4376] - Display name and Username in AD and LDAP is wrong
[FR-4386] - SDMX-JSON structures - some keys are the wrong case with respect to the schema
[FR-4397] - SDMX-JSON syntax error on contact names, departments, and roles on Organisations
[FR-4295] - UI - Dataset Details page - View data button should only view the data for the currently selected dataset
[FR-4344] - Validate SDMX-JSON data not reporting error with missing Sender ID
[FR-4349] - Unable to save a DSD with a Primary Measure which uses a Value List for one of the Components
[FR-4350] - Unable to create a DSD when there is no concept with OBS_VALUE in the Registry
[FR-4351] - Primary Measure has incorrect URN with new DSD built through UI Wizard
[FR-4352] - EDI - report error when writing output with unsupported characters
[FR-4354] - EDI file - CTA segment should not restrict contact-id and contact-name to be of IDType
[FR-4358] - Server error when modifying report template from having an Attribute in a worksheet to becoming a fixed attribute
[FR-4359] - Client side issue - unable to modify Report template Attribute from being included in worksheet to having a fixed value
[FR-4362] - Do not output EDI structures with non-1.0 versions, or references to structures with non-1.0 versions, as they are not compatible with the EDI specification
[FR-4365] - structure query, implement references=organisationscheme, currently returns HTTP 400
[FR-4366] - structure query, references=all does not return children of parents
[FR-4315] - EDI issue: Missing REL segment after FNS+Attributes is not detected
[FR-4317] - EDI issue: FTX segments should not be present in EDI deletion messages but no error reported
[FR-4318] - EDI issue: CSV segments should not be present in EDI deletion messages but no error reported
[FR-4270] - EDI time formats are not set correctly when using EDIDataReader
[FR-4224] - Can not convert data to JSON format when no dataflow selected
[FR-4225] - "304 Not Modified" no longer being returned when header includes "If-Modified-Since"
[FR-4226] - Issue with reading SDMX-CSV with very large datasets
[FR-4228] - ByteOrderMark (BOM UTF8) preventing CSV read
[FR-4238] - SDMX JSON should report error when attribute indexes are wrong
[FR-4215] - Enable a dataflow mapping to be saved which is linked to a database, even if there is no data available
[FR-4205] - ws/fusion/info/product web service to include link for current user details
[FR-4187] - When specifying "Structure" in transform call, HTTP 400 is thrown rather than HTTP 200
[FR-4155] - EDI Interchange Header - Application Reference value should accept GESMES/CB and return HTTP 200 if illegal
[FR-4158] - HTTP 200 should be returned when header of EDI file has incorrect elements (ID, preparation date, test value)
[FR-4161] - Validating an EDI file ignores the specified Structure in favour of Structure referenced in EDI file itself
[FR-4058] - Mapping to EDI output is not putting the correct organisation in the sender field
[FR-4071] - Exception generated when mapping file
[FR-4100] - SDMX 1.0 Structure file with Receiver Element with Contact Element cannot be processed by Registry
[FR-4101] - Support SDMX 1.0 'Update' Action
[FR-4057] - xsi:schemaLocation should be specifiable at the Dataset element since this is valid XML
[FR-4035] - Validating Data - requesting Invalid values for EDI file with missing dimension causes 500 error
[FR-4038] - Validation EDI Time Series should report error if range specified but first and last values are not supplied
[FR-4043] - Enforce that TIME_FORMAT is never a mandatory attribute
[FR-4044] - Mapping Dataset Attributes to Series Attributes does not produce correct output
[FR-4028] - Data Validation of EDI series with missing dimensions should report error of type "Structure"
[FR-4033] - Reverting EDI TIME_FORMAT processing
[FR-4000] - Obs Status Validator is not honouring NaN as a missing value
[FR-4002] - Data Validation - EDI files should not report error for missing TIME_FORMAT attribute
[FR-4015] - Reading EDI files does not attempt to resolve Dataflow reference
[FR-4021] - Data Validation need to report EDI missing series attributes more clearly
[FR-3884] - Validation Service - ability to skip Mandatory Attributes data validator should be provided
[FR-3886] - Validation Service - Supplying invalid time definitions should return an error report
[FR-3887] - Validating data in EDI format should behave in the same manner as Structure Specific format
[FR-3889] - Validating data in Generic Data format should behave in the same manner as the Structure Specific format
[FR-3924] - Reported legal characters for ID state that asterisk may be used, when it shouldn't be
[FR-3930] - Dataflow Map - Combination Mapping shows in Mapped Components table in step 3 of the wizard when in edit mode
[FR-3931] - Data Structure and DataFlow Map Wizard - language issue
[FR-3938] - Data Validation - converting valid / invalid series to EDI
[FR-3941] - Data Validation - invalid EDI output missing any form of value
[FR-3964] - Unable to read Excel data file is FREQ is a Dataset Attribute
[FR-3965] - Dataset Attributes are not output when output format is specified as Excel
[FR-3972] - Errors in validating EDI headers should return HTTP 200 rather than 400
[FR-3973] - If EDI Message Trailer Segment (UNT) is incorrect, error is not clear to end user
[FR-3980] - Data Validation of EDI file that has duplicate Series Attributes returns HTTP 400
[FR-3875] - Provision Agreement Wizard does not honour language correctly on step 1
[FR-3920] - Authentication Service - "Support Certificate Authentication" UI toggle control does not work
[FR-3925] - Adding a space to an Active Directory Role Mapping definition removes it
[FR-3929] - Fusion Registry Sdmx CSV Format can not be published to Registry if Dataset Attributes are included
[FR-3948] - Metadata Reports not showing Structure labels if language is not available
[FR-3953] - Data Sources Manager 'Table Names' modal has the 'Obs Table' and 'Key Table' header names transposed
[FR-3950] - Metadata Management Interface - Fix multi-page pagination on Manage Metadata
[FR-3949] - Metadata Management Interface - filters broken on manage metadata table for dimensions with numeric codes
[FR-3927] - Oracle: Error shown when starting tomcat