Data Streaming Input sources

Follow

rakshit bhargava

• May 23, 2019 14:15

Hi Everyone,

Please let me know if D3S Analyze supports real-time data streaming (Apache Kafka) as an input source or do we have to store the real-time data into HDFS or hadoop and then source data using one of the in-built nodes for HDFS within Analyze to source data?

Best,

Rakshit

0

• 

  Adrian Williams

  • June 05, 2019 09:01

  Data3Sixty Analyze does not currently provide a pre-built input connector for real-time data streaming sources such as Apache Kafka. However you may be able to leverage the Kafka Java client as part of a custom node using Analyze's Java node, or use the Kafka Python client in conjunction with Analyze's Python node.

  If you have a Kafka HDFS Sink Connector that is able to export the contents of a Kafka topic to an HDFS file then an Analyze HDFS Download node could be used to read the contents of the file. 

  0

  Comment actions Permalink
• 

  Adrian Williams

  • January 13, 2020 15:12

  The Data360 DQ+ product supports streaming data using Kafka. See the DQ+ documentation for further details:

  https://doc.infogixsaas.com/dqplus/Default.htm#h-nodes/nodes-streaming.htm 

  0

  Comment actions Permalink

Please sign in to leave a comment.