Data Validation - Revision history

Vmurrell: /* Overview */

2024-03-28T12:53:07Z

‎Overview

Vmurrell at 12:22, 17 August 2022

2022-08-17T12:22:46Z

Vmurrell at 12:08, 17 August 2022

2022-08-17T12:08:48Z

Glenn: /* Syntax Agnostic Validation */

2021-03-18T12:13:20Z

‎Syntax Agnostic Validation

Glenn: /* Overview */

2021-03-18T12:13:02Z

‎Overview

Vmurrell: /* Web Service */

2021-03-15T09:52:28Z

‎Web Service

Vmurrell: /* Performance */

2021-03-15T09:52:06Z

‎Performance

Vmurrell: /* Validation With Transformation */

2021-03-15T09:50:11Z

‎Validation With Transformation

Vmurrell: /* Syntax Agnostic Validation */

2021-03-15T09:49:52Z

‎Syntax Agnostic Validation

Vmurrell: /* Syntax Validation */

2021-03-15T09:47:26Z

‎Syntax Validation

@@ Line 3: / Line 3: @@
 = Overview =
-<p><strong>Data Validation</strong> checks principally whether a dataset conforms to the [[Data Structure Definition]] and complies with any specified [[Content Constraint|Content Constraints]].</p>
+<p><strong>Data Validation</strong> checks principally whether a dataset conforms to the Data Structure Definition and complies with any specified Content Constraints.</p>
 <p>In Fusion Metadata Registry, data validation is a three-step process:</p>

← Older revision		Revision as of 12:22, 17 August 2022
Line 1:		Line 1:
	[[Category:Concepts_Reference_V10]]		[[Category:Concepts_Reference_V10]]
		+	[[Category:Concepts_Reference_V11]]

	= Overview =		= Overview =

← Older revision		Revision as of 12:08, 17 August 2022
Line 1:		Line 1:
−	[[Category:~~Concepts Reference~~]]	+	[[Category:Concepts_Reference_V10]]

	= Overview =		= Overview =

@@ Line 66: / Line 66: @@
 <p>An conceptual example of the Validation Manager delegating validation questions to each Validation Engine in turn</p>
-[[File:Data-validation-engine.png|600px]]
+[[File:Data-validation-engine.png|400px]]
 <p>The purpose of a Validation Engine is to perform ONE type of validation, this allows configuration of each validation engine as a separate entity, and new validation engines can be easily added to the product if there is a new type of validation rule to implement. Validation Engines can be switched off, or have a different level of error reporting set, validation engines can also have a error limit set, so that a single engine can be decommissioned from validating a particular dataset if it is reporting too many errors.  In the validation report that is produced, the errors are grouped per validation engine.</p>

@@ Line 11: / Line 11: @@
 </ol>
-[[File:Data-validation-process.png|600px]]
+[[File:Data-validation-process.png|400px]]
 <p>

@@ Line 151: / Line 151: @@
 = Web Service =
-As an aternative to the Fusion Registry web User Interface [[Data_Validation_Web_Service|data validation web service]] can be used to validate data from external tools or processes.
+As an alternative to the Fusion Registry web User Interface [[Data_Validation_Web_Service|data validation web service]] can be used to validate data from external tools or processes.
 = Security =

@@ Line 105: / Line 105: @@
 = Performance =
-<p>The data format has some impact on performance time, as the time taken to perform the initial data read and syntax specific validation rules are format specific.  After the initial checks are performed, an intermediary data format is used to perform consolidation and syntax agnositc checks.  Therefore the performance of the data consolidation stage and syntax agnostic data validaiton is the same regardless of import format.</p>
+<p>The data format has some impact on performance time, as the time taken to perform the initial data read and syntax specific validation rules are format specific.  After the initial checks are performed, an intermediary data format is used to perform consolidation and syntax agnostic checks.  Therefore the performance of the data consolidation stage and syntax agnostic data validation is the same regardless of import format.</p>
 <p>Considerations to take to optimise performance is</p>
@@ Line 112: / Line 112: @@
 <li>In the case the dataset is coming from a URL support gzip response<li>
 <li>Use a fast hard drive to optimise I/O as temporary files will be used in the case of validating large datasets</li>
-<li>Performance is dependant on CPU speed</li>
+<li>Performance is dependent on CPU speed</li>
 </ul>
-<p>When the server recieves a zip file, there is some overhead in unzipping the file, but this overhead is very small compare to the performance gains in network transfer times.</p>
+<p>When the server receives a zip file, there is some overhead in unzipping the file, but this overhead is very small compare to the performance gains in network transfer times.</p>
 <p>Tests have been carried out on a dataset with the following properties</p>

@@ Line 99: / Line 99: @@
 |}
-= Validation With Transformation =
+= Validation with Transformation =
 Fusion Registry supports a validation process, which combines both data validation with data transformation.  The output can just be the valid dataset (with invalid observations removed) or both the valid dataset, and invalid dataset.

@@ Line 61: / Line 61: @@
 = Syntax Agnostic Validation =
 <p>
-Syntax Agnostic Validation is where the majority of the data validation process happens.  Like the name suggests, the validation is syntax agnostic, and therefore the same validation rules and processes are applied to all datasets, regardless of the format the data was uploaded in.</p>
+Syntax Agnostic Validation is where most of the data validation process happens.  Like the name suggests, the validation is syntax agnostic, and therefore the same validation rules and processes are applied to all datasets, regardless of the format the data was uploaded in.</p>
-<p>This validation process makes use of a single '''Validation Manager''' and multiple '''Validation Engines'''.  The validation manager walks the contents of the dataset (Series and Observations) in a streaming fasion, and as each new Series or Observation is read in, it asks the same question to each registered Validation Engine - the question is ''"is this valid?"''.</p>
+<p>This validation process makes use of a single '''Validation Manager''' and multiple '''Validation Engines'''.  The validation manager walks the contents of the dataset (Series and Observations) in a streaming fashion, and as each new Series or Observation is read in, it asks the same question to each registered Validation Engine - the question is ''"is this valid?"''.</p>
 <p>An conceptual example of the Validation Manager delegating validation questions to each Validation Engine in turn</p>
 [[File:Data-validation-engine.png|600px]]
-<p>The purpose of a Validation Engine is to perform ONE type of validation, this allows configuration of each validation engine as a seperate entity, and new validation engines can be easily added to the product if there is a new type of validation rule to implement. Validation Engines can be switched off, or have a different level of error reporting set, validation engines can also have a error limit set, so that a single engine can be decommisioned from validating a particular dataset if it is reporting too many errors.  In the validation report that is produced, the errors are grouped per validation engine.</p>
+<p>The purpose of a Validation Engine is to perform ONE type of validation, this allows configuration of each validation engine as a separate entity, and new validation engines can be easily added to the product if there is a new type of validation rule to implement. Validation Engines can be switched off, or have a different level of error reporting set, validation engines can also have a error limit set, so that a single engine can be decommissioned from validating a particular dataset if it is reporting too many errors.  In the validation report that is produced, the errors are grouped per validation engine.</p>
 <p>The following table shows each validation engine and its purpose</p>

@@ Line 18: / Line 18: @@
 = Syntax Validation =
-Syntax Validation refers to validaiton of the reported dataset in terms of the file syntax.  If the dataset is in SDMX-ML then this will ensure the XML is formatted correctly, and the XML Elements and XML Attributes are as expected.  If the dataset is in Excel Format (propriatory to the Fusion Registry) then these checks will ensure the data complies with the expected Excel format.
+Syntax Validation refers to validation of the reported dataset in terms of the file syntax.  If the dataset is in SDMX-ML then this will ensure the XML is formatted correctly, and the XML Elements and XML Attributes are as expected.  If the dataset is in Excel Format (proprietary to the Fusion Metadata Registry) then these checks will ensure the data complies with the expected Excel format.
 = Duplicates Validation =