Overview of the Spark-Streaming Integration API

The platform's Spark-Streaming Integration API provides a way for consuming stream data using the Apache Spark Streaming API. The integration API exposes a V3IOUtil object that contains a createDirectStream method for mapping platform V3IO streams to a Spark input stream. You can use the Spark input streams that you create with this method to consume record data and metadata from platform streams via the Spark Streaming API.

This section documents the platform's Spark-Streaming Integration Scala API for Spark v3.2.3, which is provided in the org.apache.spark.streaming.v3io package.