Kafka Streams with Jay Kreps
Software Engineering Daily,
Originally posted on Software Engineering Daily
Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and microservices that process data streams.”
Jay is the CEO of Confluent, a company that is building Kafka technology, and he is one of the original authors of Kafka. We began with a brief history of why Kafka evolved to be the message broker of choice for so many data engineering stacks, then we got into a discussion of streaming–what is streaming? How does Kafka communicate with streaming frameworks? Our conversation concluded with a discussion of Kafka Streams and the future of the Kafka data platform.
|Alooma is your data pipeline as a service. Alooma is a fully managed tool for pulling from different data sources–MySQL, Postgres, elasticsearch, Salesforce, and many others. Go to alooma.com/sedaily for more information.|
|The Insight Data Engineering Program helps software engineers level up their careers. Apply now at insightdataengineering.com/sedaily and skip straight to the coding challenge–a special offer for Software Engineering Daily listeners.|