Difference between revisions of "Apache Kafka integration"
(→Configuration) |
(→Structure Message Publication Behaviour) |
||
| Line 60: | Line 60: | ||
=Structure Message Publication Behaviour= | =Structure Message Publication Behaviour= | ||
| − | ==Kafka Message Key== | + | ===Kafka Message Key=== |
The SDMX Structure URN is used as the Kafka Message Key. Consumers must therefore be prepared to receive and interpret the key as the Structure URN correctly when processing the message. This is particularly important for structure deletion where the Message Key is the only source of information about which structure is being referred to. | The SDMX Structure URN is used as the Kafka Message Key. Consumers must therefore be prepared to receive and interpret the key as the Structure URN correctly when processing the message. This is particularly important for structure deletion where the Message Key is the only source of information about which structure is being referred to. | ||
| − | ==Additions and Modifications== | + | ===Additions and Modifications=== |
Addition and Modification of structures both result in an SDMX 'replace' message containing the full content of the structures. Deltas (such as the addition of a Code to a Codelist) are not supported. | Addition and Modification of structures both result in an SDMX 'replace' message containing the full content of the structures. Deltas (such as the addition of a Code to a Codelist) are not supported. | ||
| − | ==Deletions== | + | ===Deletions=== |
Deletion of structures results in a 'tombstone' message, i.e. one with null payload but the URN of the deleted structure in the Message Key. | Deletion of structures results in a 'tombstone' message, i.e. one with null payload but the URN of the deleted structure in the Message Key. | ||
| − | ==Kafka Transactions== | + | ===Kafka Transactions=== |
The addition and / or modification of a number of structures in a single Registry process are encapsulated into a Kafka Transaction, as when processing an SDMX-ML message containing more than one structure.<br> | The addition and / or modification of a number of structures in a single Registry process are encapsulated into a Kafka Transaction, as when processing an SDMX-ML message containing more than one structure.<br> | ||
Structure deletions are never mixed with additions / modifications in the same transaction. This reflects the behaviour of the SDMX REST API whereby additions / modifications use HTTP POST, while deletions require HTTP DELETE. | Structure deletions are never mixed with additions / modifications in the same transaction. This reflects the behaviour of the SDMX REST API whereby additions / modifications use HTTP POST, while deletions require HTTP DELETE. | ||
| − | ==Full Content Publication== | + | ===Full Content Publication and Re-synchronisation=== |
Under certain conditions, the Fusion Registry Producer will publish all of its structures in a single SDMX Structure Message of the chosen format to Kafka.<br> | Under certain conditions, the Fusion Registry Producer will publish all of its structures in a single SDMX Structure Message of the chosen format to Kafka.<br> | ||
These are: | These are: | ||
| Line 83: | Line 83: | ||
It also mitigates the risk that changes held in the Registry's in-memory queue of messages awaiting Kafka publication were lost on shutdown by forcing Consumers to re-synchronise on Registry startup. | It also mitigates the risk that changes held in the Registry's in-memory queue of messages awaiting Kafka publication were lost on shutdown by forcing Consumers to re-synchronise on Registry startup. | ||
| − | ==Not Supported: Subscription to Specific Structures== | + | ===Not Supported: Subscription to Specific Structures=== |
The current implementation does not allow Kafka Consumers to subscribe to changes on specific structures or sets of structures, such as those maintained by a particular Agency. Consumers subscribing to the chosen topic will receive information about all structures and will need to select what they need. | The current implementation does not allow Kafka Consumers to subscribe to changes on specific structures or sets of structures, such as those maintained by a particular Agency. Consumers subscribing to the chosen topic will receive information about all structures and will need to select what they need. | ||
| − | ==Publication Arbitration on Fusion Registry Load Balanced Clusters== | + | ===Publication Arbitration on Fusion Registry Load Balanced Clusters=== |
IMPORTANT: There is currently no arbitration mechanism between peer Fusion Registry instances configured in a Load Balanced cluster. This means that each member of the cluster independently publishes each structure change notification to Kafka, resulting in duplication of notifications. | IMPORTANT: There is currently no arbitration mechanism between peer Fusion Registry instances configured in a Load Balanced cluster. This means that each member of the cluster independently publishes each structure change notification to Kafka, resulting in duplication of notifications. | ||
Revision as of 02:30, 7 February 2020
Contents
Compatibility
| Product | Module | Version | Support |
|---|---|---|---|
| Fusion Registry Enterprise Edition | Core | 10.0.1 | Kafka producer supporting 'Structure Notification' events |
Overview
Fusion Registry can act as an Apache Kafka Producer where specified events are published on definable topics.
The library of possible events is extensible with the expection that new events can be easily added as required when use cases emerge. At present, the Kafka publication system recognises only one event, namely 'Structure Notification' which publishes changes to structures in a similar way to that of the RSS feed and the email notification subscription process.
Other events are envisaged include:
- Anything that's audited including data registrations, user login events etc
- Errors
- Configuration changes, e.g. changes to server settings
- Changes to Content Security Rules
Connection to a single Kafka service is supported.
Configuration
Configuration is performed through the GUI with 'admin' privileges.
Connection
| Paramater | Value |
|---|---|
| Client ID | A unique identifier for the Fusion Registry client. There's no real restrictions. |
| Host | Hostname or IP address of the Kafka Service as defined by the Kafka administrator |
| Port | Port number of the Kafka service |
| Comression Algorithm | The algorithm to compress the payload. Choices are: None, GZIP, Snappy, LZ4. |
| Enable Kerberos Security | If Enabled,the Producer attempts to authenticate with the specified Kerberos service for access to Kafka |
Topics
Defines the topic(s) for the event type.
Kafka Topic is ID of the topic on which to publish. The Spring Kafka interface on which the functionality is based indicates that multiple topics can be specified separated by commas.
So 'FOO' publishes on just the single topic specified.
'FOO,BAR' publishes on both FOO and BAR topics.
Note that publication on multiple topics has not been comprehensively tested.
General Behaviour
The Producer sub-system will attempt to publish events to Kafka in real time. If the publication fails, the message is placed on an in-memory queue which an independent thread periodically attempts to process. The thread is resilient and is replaced on failure to ensure queue is never left un-monitored.
IMPORTANT: The in-memory queue is not persisted so changes to Registry structures that are yet to be sucessfully published to Kafka will be lost if the Registry service is stopped.
Structure Message Publication Behaviour
Kafka Message Key
The SDMX Structure URN is used as the Kafka Message Key. Consumers must therefore be prepared to receive and interpret the key as the Structure URN correctly when processing the message. This is particularly important for structure deletion where the Message Key is the only source of information about which structure is being referred to.
Additions and Modifications
Addition and Modification of structures both result in an SDMX 'replace' message containing the full content of the structures. Deltas (such as the addition of a Code to a Codelist) are not supported.
Deletions
Deletion of structures results in a 'tombstone' message, i.e. one with null payload but the URN of the deleted structure in the Message Key.
Kafka Transactions
The addition and / or modification of a number of structures in a single Registry process are encapsulated into a Kafka Transaction, as when processing an SDMX-ML message containing more than one structure.
Structure deletions are never mixed with additions / modifications in the same transaction. This reflects the behaviour of the SDMX REST API whereby additions / modifications use HTTP POST, while deletions require HTTP DELETE.
Full Content Publication and Re-synchronisation
Under certain conditions, the Fusion Registry Producer will publish all of its structures in a single SDMX Structure Message of the chosen format to Kafka.
These are:
- When a new Kafka connection is set up, or the configuration of an existing Kafka connection is changed
- Registry startup
The principle here is to ensure that Consumers have a consistent baseline against which to process subsequent change notifications.
It also mitigates the risk that changes held in the Registry's in-memory queue of messages awaiting Kafka publication were lost on shutdown by forcing Consumers to re-synchronise on Registry startup.
Not Supported: Subscription to Specific Structures
The current implementation does not allow Kafka Consumers to subscribe to changes on specific structures or sets of structures, such as those maintained by a particular Agency. Consumers subscribing to the chosen topic will receive information about all structures and will need to select what they need.
Publication Arbitration on Fusion Registry Load Balanced Clusters
IMPORTANT: There is currently no arbitration mechanism between peer Fusion Registry instances configured in a Load Balanced cluster. This means that each member of the cluster independently publishes each structure change notification to Kafka, resulting in duplication of notifications.
Structure changes made on one instance of Fusion Registry in a cluster are replicated to all of the others through either a shared database polling mechanism, or by Rabbit MQ messaging. An instance receiving information that structures have changed elsewhere in the cluster updates its in-memory SDMX information model accordingly. The update event is however trapped by its Kafka notification processor and published as normal.
Changes are required to the Kafka event processing behaviour of Registries operating in a cluster to arbitrate over which instance publishes the notification on Kafka such that a structure change is notified once and only once, as expected.
