Support and Certification Matrix
Overview
The platform comes pre-deployed with proprietary and third-party open-source tools and libraries that are exposed as application services that are managed using Kubernetes. Relevant services can be viewed and managed by users from the platform dashboard using a self-service model. (Note that some services that don't require user intervention aren't visible in the dashboard.) Users can also enhance their development experience by independently installing additional software and run it on top of the platform services. For more information, see The Platform's Application Services.
The platform has two types of managed application services:
- Default services
- There are several service instances — such as Trino and the web APIs — that are spawn automatically when the platform starts and have a tenant-wide scope (i.e., they're accessible to all tenant users with service permissions). The default services can't be deleted by users, but service administrators can disable or restart these services and modify some service configurations.
- User-defined services
- Service administrators can create a wide variety of new service instances for certified services — such as Spark and Jupyter Notebook. Except where otherwise specified, user assigned services should be assigned to a specific running user but can optionally be shared also with all other tenant users with service permissions.
Pre-deployed Application Services and Tools
The following software packages, services, and tools are pre-deployed as part of the default version 3.6.1 platform installation:
Conda | Dashboard | Docker Registry | Log forwarder | Frames | Grafana | Hadoop | Hive | Horovod | Jupyter | Kubernetes | Log Forwarder | MLRun | Monitoring | MPI Operator | Nuclio Serverless Framework | Operating System | Pipelines | Trino | OAuth2 (OIDC) Authenticator | Spark | Spark Operator | V3IO Daemon | Web APIs | Web Shell
- Open-source tools and related services are subject to open-source restrictions.
- See Application Library Versions for programming application library versions that are used or certified for usage by relevant services, such as Nuclio and Jupyter Notebook.
Service | Type | Description | Version | |
---|---|---|---|---|
Dashboard | Default | The platform's graphical user interface. | 3.6.1 | |
Kubernetes | Default | The Kubernetes (k8s) container orchestration system for automating deployment, scaling, and management of containerized applications. Application services in the platform run on top of Kubernetes. | 1.28 | |
Kubernetes on cloud | Default | The managed Kubernetes (k8s) for AKS, EKS, and GKE. | 1.28, 1.30 | |
Nuclio Serverless Framework ( nuclio ) |
Default | Iguazio's Nuclio Enterprise Edition serverless framework for development, deployment, and execution of serverless functions for real-time data processing and analysis.
The Nuclio dashboard is available as part of the |
1.12.5 | |
MLRun ( mlrun ) |
Default | Iguazio's https://github.com/mlrun/mlrun/tree/release/v1.6.x/ open-source AI orchestration platform quickly building and managing continuous (gen) AI applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications. For more information, see Data Science Automation and AI Services. | 1.6 | |
Pipelines ( pipelines ) |
Default | The Google Kubeflow Pipelines open-source framework for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers. For more information, see Data Science Automation and AI Services. | 1.8.1 | |
Web APIs ( webapi ) |
Default | The platform's web-APIs (web-gateway) service, which provides access to its web APIs. | 3.6.1 | |
Jupyter | User-defined | The JupyterLab UI, including the Jupyter Notebook web application and shell terminals and the Conda binary package and environment manager. For more information, see The Jupyter Notebook Service. See also the Jupyter application-libraries compatibility matrix. | JupyterLab 3.5.6 | |
V3IO Frames ( framesd ) |
User-defined | The platform's V3IO Frames service, which provides access to the Frames API — an open-source unified high-performance Python DataFrame API for accessing NoSQL, stream, and time-series data in the platform's data store. | 0.10 (server)
0.10 (supported client) |
|
Spark | User-defined | The Apache Spark data-processing engine, including the following libraries:
|
3.2.3 | |
Spark Operator ( spark-operator ) |
Default | The spark-on-k8s-operator Kubernetes Operator for Spark ("Spark Operator"), which enables simplifying submission and scheduling of Spark jobs. This service is designed to be used via the MLRun Spark Operator API. | 3.2.3 | |
Trino ( Trino ) |
Default | The Trino distributed SQL query engine for big data. | 370 | |
Hive Metastore | Internal | An internal Apache Hive Metastore service that can be enabled for the Trino service to allow saving views and using the Trino Hive connector. | 0.8.1 | |
Horovod / MPI Operator ( mpi-operator ) |
Default | Distributed training using Kubeflow MPI Operator and Uber's Horovod distributed deep-learning framework for creating machine-learning models that are trained simultaneously over multiple GPUs or CPUs. For more information, see The MPI-Operator Horovod Service and Running Applications over GPUs. | 0.2.3 | |
Log Forwarder ( log-forwarder ) |
Default | A platform service that uses Filebeat to forward application-service logs to be stored and indexed in an instance of the Elasticsearch search and analytics engine.
Note that this default service is disabled by default because you need to configure the URL of an Elasticsearch service for storing and indexing the logs. For more information, see Logging, Monitoring, and Debugging. |
7.9.2
Supports Elasticsearch 7.17 |
|
Monitoring ( monitoring ) |
Default | A platform service for monitoring application services and gathering performance statistics and additional data. The gathered data is visualized on Grafana dashboards using the platform's Grafana services. For more information, see Monitoring Platform Services. | 3.7 | |
Grafana | User-defined | The Grafana analytics and monitoring platform.
In cloud platform environments, Grafana is currently available as a shared single-instance tenant-wide service. The platform also has a shared single-instance tenant-wide application-cluster Grafana service with monitoring dashboards for the entire Kubernetes application cluster, which isn't visible on the |
9.2.15 | |
Docker Registry | Default and Custom User | A platform service for working with a Docker Registry, which is used by the MLRun and Nuclio services to store the function images.
Operational clusters require a Custom User Docker Registry. Configure it in your cluster either during installation or post-install.
Playground environments can use the default on-cluster Docker Registry. (The Nuclio service in the default tenant is configured, by default, to work with a pre-deployed default tenant-wide docker-registry service that uses a pre-deployed local on-cluster Docker Registry.) |
2.8.1 | |
OAuth2 (OIDC) Authenticator ( authenticator ) |
Default | A federated OpenID Connect (OIDC) provider over OAuth2, using OpenID Connect (OIDC). This service is used for OAuth2 authentication of user access to Nuclio API gateways and shared Grafana services, including access by external (non-platform) users. | 2.23.0 | |
Web Shell | User-defined | A platform service that provides a web-based command-line shell ("web shell") for running application services — such as Spark jobs and Trino queries — and performing basic file-system operations.
Note that this isn't a fully functional Linux shell. For more information, see The Web-Shell Service. |
3.6.1 | |
Hadoop | Default | The Apache Hadoop distributed data-processing library. For more information, see The Hadoop Service. | 3.3.4 | |
V3IO Daemon ( v3io-daemon ) |
Default (internal) | An internal service for integrating the platform with external applications by using the platform's V3IO library. | 3.6.1 | |
TensorBoard |
Default | TensorFlow's Tensorboard visualization toolkit. Tech Preview | 3.6.1 | |
Operating System | Internal | The Rocky Linux operating-system. | 8.x |
Application Library Versions
The following table provides information about the versions of application libraries (packages) that are used or certified for usage with different pre-deployed platform tools and services:
Service | Pre-deployed and Certified Application Library Versions |
---|---|
Platform API Libraries | Java 8
Scala .7 and 2.11 |
Frames | Python 3.7 |
MLRun | Python 3.9 |
MlRun | Python 3.9 |
Nuclio | Python 3.9 |
Jupyter | Iguazio platform tutorials (v3io/tutorials) 3.5 Iguazio V3IO Python SDK (v3io/v3io-py) 0.6 (Python 3.6–3.8) Nuclio Jupyter package (nuclio/nuclio-jupyter) 0.9 Python 3.7 Conda 4.8.3 NVIDIA CUDA 11.0 NVIDIA RAPIDS 0.17 Frames client 0.10 |
Web Shell | Python 3.9
Scala 3.9 |
Integration with Additional Tools
You can independently install additional software tools — such as TensorFlow, PyTorch, or scikit-learn — and use them on top of the platform services. You can also configure remotely installed tools — such as Tableau or Looker — to analyze and visualize data in the platform. In addition, you can use Conda (which is available as part of the platform's Jupyter Notebook service) and pip (which is available as part of the Jupyter Notebook, and web-shell services) to install Python packages.
For more information, see The Platform's Application Services.