Apache Kafka integration - Revision history

Vmurrell at 12:11, 26 March 2024

2024-03-26T12:11:18Z

2024-03-23T12:06:04Z

2022-09-05T13:37:34Z

2022-08-25T13:02:11Z

‎Kafka Message Body

2022-08-25T13:01:08Z

‎Kafka Message Body

2021-05-03T09:04:01Z

‎Structure Notification

2021-05-03T09:02:59Z

‎Setting Up

2021-05-03T09:02:14Z

‎Test Connection

2021-02-23T16:26:48Z

‎Overview

2021-02-23T12:05:31Z

@@ Line 1: / Line 1: @@
-[[Category:Draft_-_Installation_and_Configuration]]
+[[Category:Installation_and_Configuration]]
 [[Category:FMR_Configuration_Reference]]
 [[Category:How_To]]

← Older revision		Revision as of 12:06, 23 March 2024
Line 1:		Line 1:
		+	[[Category:Draft_-_Installation_and_Configuration]]
		+
	[[Category:FMR_Configuration_Reference]]		[[Category:FMR_Configuration_Reference]]
	[[Category:How_To]]		[[Category:How_To]]

@@ Line 1: / Line 1: @@
 [[Category:FMR_Configuration_Reference]]
 [[Category:How_To]]
 =Overview=
 The Kafka Integration function is available to logged-in users and found on on the Main menu.

@@ Line 194: / Line 194: @@
 |}
-  Events for certain structures will only be published if the structure type is supported by the chosen format. Only Fusion-JSON supports all possible structures including Fusion Metadata Registry 'extended structures' like [[Validation scheme|Validation_Scheme]].
+  Events for certain structures will only be published if the structure type is supported by the chosen format. Only Fusion-JSON supports all possible structures including Fusion Metadata Registry 'extended structures' like [[Validation_Scheme|ValidationScheme]].
 ===Additions and Modifications===

@@ Line 194: / Line 194: @@
 |}
-  Events for certain structures will only be published if the structure type is supported by the chosen format. Only Fusion-JSON supports all possible structures including Fusion Metadata Registry 'extended structures' like [[Validation scheme|Validation Schemes]].
+  Events for certain structures will only be published if the structure type is supported by the chosen format. Only Fusion-JSON supports all possible structures including Fusion Metadata Registry 'extended structures' like [[Validation scheme|Validation_Scheme]].
 ===Additions and Modifications===

@@ Line 167: / Line 167: @@
-[[File:KafkaMetadataNotification.png||700px]]
+[[File:KafkaMetadataNotification.png|Kafka Model Notification|700px]]
 ===Kafka Message Key===
@@ Line 233: / Line 233: @@
 Changes are required to the Kafka event processing behaviour of Registries operating in a cluster to arbitrate over which instance publishes the notification on Kafka such that a structure change is notified once and only once, as expected.
 = Registry Database Tables =

@@ Line 104: / Line 104: @@
 ==== Setting Up ====
 [[File:Kafka structure producer.PNG|800px]]
 The Structure Producer uses the Kafka connection to send all structural changes. However, before the structure producer becomes operational, it must be assigned to the Kafka topic to use, and must be given a message format to notify structures in.

@@ Line 90: / Line 90: @@
 Example connection test results
-[[File:Kafka test results.PNG|400px]]
+[[File:Kafka test results.PNG|Test Kafka Connection|400px]]
 The Fusion Metadata Registry ensures the bootstrap servers can all be connected to before the connection is saved.  However, the test function should be used to ensure the following:

@@ Line 2: / Line 2: @@
 [[Category:How_To]]
 =Overview=
 Fusion Metadata Registry can act as an Apache Kafka Producer where specified events are published on definable Kafka [https://kafka.apache.org/quickstart#quickstart_createtopic topics].
 It consists of a generalised ‘producer’ interface capable of publishing any information to definable Kafka topics, and a collection of ‘handlers’ for managing specific events.

@@ Line 158: / Line 158: @@
 ==Structure Notification==
-SDMX Structural Metadata is published to definable Kafka topics on Fusion Metadata Registry startup and each time structures are added / modified, or deleted. Metadata-driven applications can subscribe to the relevant topic(s) to receive notifications of changes to structures allowing them to maintain an up-to-date replica copy of the structural metadata they need.
+SDMX Structural Metadata is published to definable Kafka topics on Fusion Metadata Registry start-up and each time structures are added / modified, or deleted. Metadata-driven applications can subscribe to the relevant topic(s) to receive notifications of changes to structures allowing them to maintain an up-to-date replica copy of the structural metadata they need.
 Example use cases include:
@@ Line 235: / Line 235: @@
 = Registry Database Tables =
-The folowing database tables are used in the Fusion Registry to store Kafka information
+The following database tables are used in the Fusion Registry to store Kafka information