Apache Kafka integration
Contents
Compatibility
| Product | Module | Version | Support |
|---|---|---|---|
| Fusion Registry Enterprise Edition | Core | 10.0 | Kafka producer supporting 'Structure Notification' events |
Overview
Fusion Registry can act as an Apache Kafka Producer where specified events are published on definable topics.
The library of possible events is extensible with the expection that new events can be easily added as required when use cases emerge. At present, the Kafka publication system recognises only one event, namely 'Structure Notification' which publishes changes to structures in a similar way to that of the RSS feed and the email notification subscription process.
Other events are envisaged include:
- Anything that's audited including data registrations, user login events etc
- Errors
- Configuration changes, e.g. changes to server settings
- Changes to Content Security Rules
Connection to a single Kafka service is supported.
Configuration
Configuration is performed through the GUI with 'admin' privileges.
Connection
| Paramater | Value |
|---|---|
| Client ID | A unique identifier for the Fusion Registry client. There's no real restrictions. |
| Host | Hostname or IP address of the Kafka Service as defined by the Kafka administrator |
| Port | Port number of the Kafka service |
| Comression Algorithm | The algorithm to compress the payload. Choices are: None, GZIP, Snappy, LZ4. |
| Enable Kerberos Security | If Enabled,the Producer attempts to authenticate with the specified Kerberos service for access to Kafka |
Topics
Defines the topic(s) for the event type.
Kafka Topic is ID of the topic on which to publish. The Spring Kafka interface on which the functionality is based indicates that multiple topics can be specified separated by commas.
So 'FOO' publishes on just the single topic specified.
'FOO,BAR' publishes on both FOO and BAR topics.
Note that publication on multiple topics has not been comprehensively tested.
General Behaviour
Publication Reliability
Message publication is not 100% reliable.
Fusion Registry places event messages as they are created onto an internal in-memory Staging Queue. When a message is created, an immediate attempt is made to publish to the Kafka broker cluster. If the publication fails (the Kafka service is unavailable, for instance), the message is returned to the Queue. An independent Queue Processor thread makes periodic attempts to complete the publication of staged messages.
IMPORTANT: Messages stay on the Staging Queue until they are successfully published to Kafka. However, the Staging Queue is not persistent so any staged but unpublished messages will be lost when the Fusion Registry service terminates.
So messages will be lost in the case where a Fusion Registry instance is unable to publish staged messages to Kafka before it is terminated. Each individual event message service manages the risk as appropriate. For Structure Notification, the risk is managed by forcing consumers to re-synchronise their complete structural metadata content with the Registry on Registry startup. Even if structure change event messages were lost, Consumers' metadata is reset to a consistent state.
Events
Structure Notification
Kafka Message Key
The SDMX Structure URN is used as the Kafka Message Key. Consumers must therefore be prepared to receive and interpret the key as the Structure URN correctly when processing the message. This is particularly important for structure deletion where the Message Key is the only source of information about which structure is being referred to.
Additions and Modifications
Addition and Modification of structures both result in an SDMX 'replace' message containing the full content of the structures. Deltas (such as the addition of a Code to a Codelist) are not supported.
Deletions
Deletion of structures results in a 'tombstone' message, i.e. one with null payload but the URN of the deleted structure in the Message Key.
Kafka Transactions
The addition and / or modification of a number of structures in a single Registry process are encapsulated into a Kafka Transaction, as when processing an SDMX-ML message containing more than one structure.
Structure deletions are never mixed with additions / modifications in the same transaction. This reflects the behaviour of the SDMX REST API whereby additions / modifications use HTTP POST, while deletions require HTTP DELETE.
Full Content Publication and Re-synchronisation
Under certain conditions, the Fusion Registry Producer will publish all of its structures in a single SDMX Structure Message of the chosen format to Kafka.
These are:
- When a new Kafka connection is set up, or the configuration of an existing Kafka connection is changed
- Registry startup
The principle here is to ensure that Consumers have a consistent baseline against which to process subsequent change notifications.
It also mitigates the risk that changes held in the Registry's in-memory queue of messages awaiting Kafka publication were lost on shutdown by forcing Consumers to re-synchronise on Registry startup.
Not Supported: Subscription to Specific Structures
The current implementation does not allow Kafka Consumers to subscribe to changes on specific structures or sets of structures, such as those maintained by a particular Agency. Consumers subscribing to the chosen topic will receive information about all structures and will need to select what they need.
Publication Arbitration on Fusion Registry Load Balanced Clusters
IMPORTANT: There is currently no arbitration mechanism between peer Fusion Registry instances configured in a Load Balanced cluster. This means that each member of the cluster independently publishes each structure change notification to Kafka, resulting in duplication of notifications.
Structure changes made on one instance of Fusion Registry in a cluster are replicated to all of the others through either a shared database polling mechanism, or by Rabbit MQ messaging. An instance receiving information that structures have changed elsewhere in the cluster updates its in-memory SDMX information model accordingly. The update event is however trapped by its Kafka notification processor and published as normal.
Changes are required to the Kafka event processing behaviour of Registries operating in a cluster to arbitrate over which instance publishes the notification on Kafka such that a structure change is notified once and only once, as expected.
