We recommend switching to the latest versions of Edge, Firefox, Chrome or Safari. Using Internet Explorer will result in a loss of website functionality.

Kafka Message Validation and Reconciliation

Comments

3 comments

  • Avatar
    Jeffery Brown

    For those interested in Kafka, we also have a few new articles on the Infogix website:

    - https://www.infogix.com/defending-data-quality-in-a-real-time-data-world/

    - https://www.infogix.com/resources/data3sixty-for-kafka-data-sheet/

    0
    Comment actions Permalink
  • Avatar
    Rob McCurley

    My organization has a new administrative system that exports data using Kafka. We'd like to be able to directly consume these Kafka topics/streams in DQ+ to do things like data quality checks. Until this functionality appears in DQ+ we'll  probably need to stream data to a file data source which will be read and processed on a scheduled basis. A couple of features that seem like they would be needed are:

    • Ability to trigger execution of an analysis or process model directly from the Kafka topic.
    • Ability to control how often the analysis or process model is run: once for each message, only after a specified number of messages have accumulated, on a timed basis, or some combination of all of these. 

    However, we don't have a well-developed set of requirements right now other than the need to ingest data from Kafka. 

    0
    Comment actions Permalink
  • Avatar
    Christopher Reed

    Rob,

    Thanks for the post.  We currently have the capabilities you describe available in tech preview mode in the on premise version of the solution.  Most of our customers are in the early stages of gathering requirements for Kafka. As more details become available we will be sharing them on the site.

    Thanks.

    0
    Comment actions Permalink

Please sign in to leave a comment.