| Sign up
Marketplace
Kafka Incremental Read in a CDI Mapping

Kafka Incremental Read in a CDI Mapping

Posted by: Informatica Success Portal

Free

This sample Mapping demonstrates how one can use a normal CDI Mapping to perform an Incremental read from a Kafka topic.

Data Integration
Download Now

Overview

The attached resource demonstrates how one can use a normal CDI Mapping to perform an Incremental read from a Kafka Topic (from the 2nd run onwards) and all messages in the first run.
It achieves this by making use of two InOut Parameters along with the timestamp available in a Kafka Message.

  1. Each Kafka Message has a timestamp field made available to us in the Source Transformation (Image 1)
  2. Additionally, the Source Transformation allows us to filter messages from the Kafka Topic from a certain timestamp (image 2)
  3. Taking advantage of these two aspects, the logic is as follows:
    1. In the current run, we store the timestamp of each message by updating the In-Out Parameter (LastMessageTimestamp ). At the end of the mapping run , this parameter will have the timestamp of the Last Message in the current run
    2. In the next, run, this parameter will be used in Source to Filter messages, which will lead to us reading the LAST MESSAGE from the previous run + Any new messages in the topic. This is necessary as the topic may have received messages within a gap of nanoseconds (OR) the same nanosecond as well after the previous message
  4. To avoid issues, we read the last message from the previous run, but we filter it out downstream using the combination of a Sequence Generator and a Filter Transformation.
  5. To handle the scenario, where this mapping is being run for the first time, we have an additional control flag named “FirstTime” which will be set to 0 initially and from the second run will be automatically set to 1 from the next run. This is used in the filter along with the sequence number to decide if the first record should be sent downstream (Which is needed in the first run).

Resources

Attached resource kit contains the CDI Mapping export that demonstrates Incremental read from a Kafka topic

Support

For any questions or clarifications, please contact Informatica Global Customer Support.