Description

Kafka Streams API is an advanced API that’s part of the Kafka Ecosystem.

Using Kafka Streams API, we can :

    Apply data transformations,
    Data enrichment,
    Branching the data into multiple data streams.
    Aggregating the data or joining data from multiple Kafka topics.
    Aggregate the Data into Window Buckets and more.

The Kafka Streams API For Developers using Java/SpringBoot course is structured to give you both the theoretical and coding experience of developing Kafka Streams Applications using Streams API and also covers the techniques to use Enterprise Standard Kafka Streams Application using SpringBoot and Streams API .

This is a pure hands-on oriented course where you will be learning the concepts through code. You will build a realtime Kafka Streams application by the end of this course.

By the end of this course, you will have a complete understanding of these concepts:

    Building Kafka Streams Applications using Streams API
    Building Kafka Streams Applications using SpringBoot & Streams API
    Write Interactive Queries to retrieve the aggregated data from a state store and expose it via RESTFUL API.
    Unit and Integration Testing Kafka Streams Applications using JUnit5

Getting Started to Kafka Streams

    In this section, I will give you all an introduction to Kafka streams and the different terminologies that are involved in build a Kafka Streams Application.
        Introduction to Kafka Streams
        Kafka Streams Terminologies – Topology & Processor
        Introduction to KStreams API

Greetings Kafka Streams App using KStreams API

In this section, we will build a simple Kafka Streams App and test it in our local.

    Learn to build a Topology of the Greetings App
    Build the Kafka Streams Launcher Application using which we can start and stop the application.

Operators in Kafka Streams using KStream API

In this section, we will explore some of the operators in the thats part of the Kafka Streams API.

    Filter & FilterNot
    Map/MapValues
    FlatMapValues/FlatMap
    peek
    merge

Serialization and Deserialization in Kafka Streams

In this section, we will code and explore the serialization and deserialization in Kafka Streams.

    How Key/Value serialization and deserialization works in Kafka Streams ?
    Providing Default Serializer/Deserializer using Application Configuration
    Build a Custom Serdes for Enhanced Greeting Messages

Reusable Generic Serializer/Deserializer (Recommended Approach)

In this section, I will show you the best approach to build a Generic Serializer and Deserializer that can be used for any type of Messages.

    Build a Generic Serializer/Deserializer

Order Management Kafka Streams application – A real time use case

In this section, we will build a kafka streams application by implementing a Order Management system for a retail company

Topology, Stream and Tasks – Under the Hood

In this section, we will explore the internals of the Kafka Streams Application.

    Internals of Topology, Stream and Tasks

Error/Exception Handling in Kafka Streams

In this section, we will explore different error handlers in Kafka Streams.

    Failures in Kafka Streams
    Default Deserialization Error Behavior
    Custom Deserialization Error Handler
    Default & Custom Processor Error Handler
    Custom Production Error Handler

KTable & Global KTable

In this section, we will explore the KTable and GlobalKTable in KafkaStreams.

    Introduction to KTable API
    Build a topology for KTable
    KTable – Under the Hood
    GlobalKTable

StateFul Operations in Kafka Streams – Aggregate, Join and Windowing Events

In this section, I will give an introduction to stateful operators in Kafka Streams and explore the aggregation related operators in Kafka streams.

    StateFul Operations in Kafka Streams
    How aggregation works ? & Aggregation using “count” operator
    Group Records by using groupBy operator
    Aggregation using “reduce” operartor
    Aggregation using “aggregate” operator
    Using Materialized views for count & reduce operator

StateFul Operation Results – How to access them ?

In this section, I will explain about the options to retrieve the results of the aggregation.

Re-Keying Kafka Records for Stateful operations

In this section, we will code and explore the effect of null operator and the need to rekeying records during stateful operations.

StateFul Operations in Kafka Streams – Join

In this section, we will code and explore the different types of Joins in Kafka Streams Application.

Join in Order Management Application – A Real Time Use Case

In this section, we will implement join in the order management application that we have been working on so far.

    Introduction to Joins & Types of Joins in Kafka Streams
    Explore innerJoin using “join” operator – Joining KStream and KTable
    Explore innerJoin using “join” operator – Joining KStream and GlobalKTable
    Explore innerJoin using “join” operator – Joining KTable and KTable
    Explore innerJoin using “join” operator – Joining KStream and KStream
    Joining Kafka Streams using “leftJoin” operator
    Joining Kafka Streams using “outerJoin” operator
    Join – Under the hood
    CoPartitioning Requirements in Joins & Joins Under the Hood

StateFul Operations in Kafka Streams – Windowing

In this section, we will explore the windowing concepts in Kafka Streams.

    Introduction to Windowing and time concepts
    Windowing in Kafka Streams – Tumbling Windows
    Control emission of windowed results using “supress” operartor
    Windowing in Kafka Streams – Hopping Windows
    Windowing in Kafka Streams – Sliding Windows

Widowing in Order Management Application – A Real Time Use Case

In this section, we will code and explore the new requirement to implement the windowing in the Orders Stream Application.

Behavior of Records with Future & Older Timestamp in Windowing

In this section, we will explore the behavior of records with the older and future timestamp in a Kafka Streams Application.

    Build Kafka Streams Application using SpringBoot
    In this section, we will build a simple kafka streams app using SpringBoot.
    Introduction to SpringBoot and Kafka Streams
    Setup the Project – Greeting Streams app using Spring Kafka Streams
    Configuring the Kafka Stream using application.yml
    Build the Greeting Topology
    Test Greeting App in Local

SpringBoot AutoConfiguration of Kafka Streams

In this section, I will show you how spring boot auto configures Kafka Streams Application.

JSON Serialization/Deserialization in Spring Kafka Streams

In this section, we will implement the JSON Serialization/Deserialization in Kafka Streams using SpringBoot.

Error Handling in Spring Kafka Streams

In this section, I will show you error handling in Kafka Streams using SpringBoot.

    Handle DeSerialization Error – Approach 1
    Handle DeSerialization Error using Custom Error Handler – Approach 2
    Handle Deserialization Error – Approach 3 ( Spring Specific Approach)
    Handle UncaughtExceptions in the Topology
    Handle Production Errors

Build Orders Kafka Streams Application using SpringBoot

In this section, we will set up the Spring Boot Project for orders streams.

Interactive Queries – Querying State Stores using RESTFUL APIs

    Build a GET Endpoint to retrieve the OrderCount by OrderType – Part 1
    Build a GET Endpoint to retrieve the OrderCount by OrderType – Part 2
    Retrieve OrderCount by OrderType & LocationId
    Build a GET Endpoint to retrieve the OrderCount for All OrderTypes
    Build a GET Endpoint to retrieve the Revenue by OrderType
    Global Error Handling for useful Client Error Messages

Interactive Queries – Querying Window State Stores using RESTFUL APIs

    Build a GET Endpoint to Retrieve OrderCount by OrderType
    Build a GET Endpoint to Retrieve the windowed OrderCount for All OrderTypes
    Build a GET endpoint to retrieve the windowed OrderCount within a Time Range
    Build a GET Endpoint to retrieve the Revenue by OrderType

Testing Kafka Streams Using TopologyTestDriver & JUnit5

In this section, we will code and learn about how to write automated tests for Kafka Streams app.

    Testing Kafka Streams using TopologyTestDriver
    Unit Testing Greetings App – Writing Data to a Output Topic
    Unit Testing Greetings App – Testing Multiple Messages
    Unit Testing Greetings App – Error Scenario
    Unit Testing OrdersCount – Writing Data to a State Store
    Unit Testing OrdersRevenue – Writing Data to a State Store
    Unit Testing OrdersRevenue By Windows – Writing Data to a State Store
    Limitations of TopologyTestDriver

Testing Kafka Streams in SpringBoot Using TopologyTestDriver & JUnit5

In this section, we will code and learn how to write unit tests in our Kafka Streams application that’s build using SpringBoot.

Integration Testing Spring KafkaStreams App using @EmbeddedKafka

In this section, we will code and learn about writing integration tests for the Kafka Streams app using EmbeddedKafka.

    Introduction & SetUp Integration Test
    Integration Test for OrdersCount
    Integration Test for OrdersRevenue
    Integration Test for OrdersRevenue By Windows

Grace Period in Kafka Streams

In this section I will explain the concept of grace period and how it can be used in kafka streams application.

Build and Package the SpringBoot App as an Executable

In this section, we will package the kafka streams app as an executable and launch the app.

By the end of this course you will have a complete understanding of Kafka Streams API and the different kinds of applications that can be built using Kafka Streams API.
Who this course is for:

    Advanced Java Developers
    Kafka Developers who are curious to learn Kafka Streams API
    Kafka Developers who are interested in building advanced streaming applications
    Developers who wish to learn the techniques to test Kafka Streams Application using TopologyTestDriver

Requirements

    Java Knowledge is required
    Prior experience building Kafka Applications
    Prior experience working with IntelliJ or any other IDEA
    Java 17 is required
    Gradle or Maven Knowledge is needed

Last Updated 3/2023