How to Deploy an MLRun Project in a CI/CD Process with Jenkins Pipeline 

In this article, we will walk you through steps to run a Jenkins server in docker and deploy the MLRun project using Jenkins pipeline. 

Before we dive into the actual set up, let’s have a brief background on the MLRun and Jenkins. 

What is MLRun? 

MLRun is a popular open source MLOps orchestration framework that streamlines machine learning projects from data collection, experimentation, model development, feature creation, production model serving deployment and model monitoring, and the full lifecycle management of machine learning. Iguazio MLOps Platform is built with MLRun at its core, with added enterprise features such as data management, user management, real-time model serving, security, autoscaling, high availability and more. Please see mlrun and the Iguazio MLOps Platform for more information.  

Try MLRun For Yourself

Learn more about MLRun & how to quickly build and managing continuous ML and generative AI applications across their lifecycle.

Learn More

What is Jenkins Pipeline? 

A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Every change to your software (committed in source control) goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built software through multiple stages of testing and deployment. 

In a traditional build process, the code and build process are separated, code is in the SCM system while the build process is residing inside the Jenkins configuration in a Jenkins server. This process had a lot of management challenges. As we are moving towards CI/CD “as code”, more tools are provided for this new process, such as GitHub Actions, Gitlab CI/CD, Jenkins Pipeline, and more.  

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. Pipeline provides an extensible set of tools for modeling simple-to-complex delivery pipelines "as code" via the Pipeline domain-specific language (DSL) syntax. For more information about Jenkins Pipeline, please refer to this link. 

In this post, we will be looking into integration of MLRun with Jenkins pipeline. 

Prerequisites and Resources 

To deploy a MLRun project to a MLOps cluster, you need access to an Iguazio cluster 

To run a Jenkins server in docker you will need a Docker Desktop running on your desktop. This post uses the Docker Desktop on a Mac OS system as an example. Please follow docker desktop installation instructions here. 

Run Jenkins in docker 

Jenkins has very comprehensive instructions on how to set up and run Jenkins in docker. However, I’ve run into some version issues. Here I am including steps that worked in my environment and it may be convenient for you to set up a working environment quickly. 

1. Create a bridge network in Docker 

docker network create jenkins 

2. Download and run the docker:dind 

docker run \ 
  --name jenkins-docker \ 
  --rm \ 
  --detach \ 
  --privileged \ 
  --network jenkins \ 
  --network-alias docker \ 
  --env DOCKER_TLS_CERTDIR=/certs \ 
  --volume jenkins-docker-certs:/certs/client \ 
  --volume jenkins-data:/var/jenkins_home \ 
  --publish 2376:2376 \ 
  --publish 3000:3000 \ 
  docker:dind \ 
  --storage-driver overlay2 

3. Create Dockerfile with jenkins 2.32.3 and jenkins plugins 

FROM jenkins/jenkins:2.332.3-jdk11 
USER root 
RUN apt-get update && apt-get install -y lsb-release 
RUN curl -fsSLo /usr/share/keyrings/docker-archive-keyring.asc \ 
  https://download.docker.com/linux/debian/gpg 
RUN echo "deb [arch=$(dpkg --print-architecture) \ 
  signed-by=/usr/share/keyrings/docker-archive-keyring.asc] \ 
  https://download.docker.com/linux/debian \ 
  $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list 
RUN apt-get update && apt-get install -y docker-ce-cli 
USER jenkins 
RUN jenkins-plugin-cli --plugins "blueocean:1.25.3 docker-workflow:1.28" 

4. Build a new docker image 

docker build -t myjenkins-blueocean:2.332.3-1 . 

5. Run the Jenkins in docker 

docker run \ 
  --name jenkins-blueocean \ 
  --rm \ 
  --detach \ 
  --network jenkins \ 
  --env DOCKER_HOST=tcp://docker:2376 \ 
  --env DOCKER_CERT_PATH=/certs/client \ 
  --env DOCKER_TLS_VERIFY=1 \ 
  --publish 8080:8080 \ 
  --publish 50000:50000 \ 
  --volume jenkins-data:/var/jenkins_home \ 
  --volume jenkins-docker-certs:/certs/client:ro \ 
  --volume "$HOME":/home \ 
  myjenkins-blueocean:2.332.3-1 

Jenkins setup wizard 

Before you can access Jenkins, there are a few quick “one-off” steps you’ll need to perform. 

Browse to http://localhost:8080

Display the Jenkins console log: 

docker logs jenkins-blueocean 

3. Copy the automatically-generated password  

4. Unlock Jenkins with password from last step 

5. Follow the instruction to create an Admin user 

6. Restart and log in to Jenkins with the user/password 

Create a MLRun Project Demo Repo on GitHub 

1. Login to your GitHub account 

2. Create a git repo with a name (e.g. mlrun-jenkins) from this project-demo template 

Configure the Credentials in Jenkins 

1. Have your GitHub access token ready. If you don’t have one, please refer to the GitHub documentation on how to generate your GitHub access token. 

2. Browse to http://localhost:8080 

3. Log in to Jenkins 

4. Click on “Manage Jenkins” -> “Manage Credentials” ->”Store scoped to Jenkins” -> “Global credentials” -> “Add Credentials”  

5. For the kind, select “Secret text” and filling your GitHub token as Secret, and enter an ID and Description 

6. Add the V3IO_ACCESS_KEY as a credential the same way as for GitHub token 

7. Add the API key for Slack notification (please refer to Slack doc) 

Now you've completed all the necessary configuration steps for Jenkins to run a pipeline. In the following section, we will walk through the steps to run a Jenkins pipeline. 

Create a Pipeline Project in Jenkins 

1. Browse to http://localhost:8080 

2. Log in to Jenkins 

3. Click on “New Item” 

4. Enter an item name, e.g. mlrun-jenkins-project-demo 

5. Chose “Pipeline”

6. Enter the git repo info for the pipeline

7. Enter the “Jenkinsfile” to the Script Path 

8. Save the project 
    

The following is the Jenkinsfile from the example project-demo. 

pipeline { 

  agent any 

   environment { 

     RELEASE='1.0.0' 

     PROJECT_NAME='project-demo' 

   } 

  stages { 

     stage('Audit tools') { 

        steps{ 

           auditTools() 

        } 

     } 

     stage('Build') { 

           environment { 

              MLRUN_DBPATH='https://mlrun-api.default-tenant.app.aaa-spark-fs.iguazio-cd2.com' 

              V3IO_ACCESS_KEY=credentials('V3IO_ACCESS_KEY') 

              V3IO_USERNAME='xingsheng' 

           } 

           agent { 

               docker { 

                   image 'mlrun/mlrun:1.0.0' 

               } 

           } 

           steps { 

              echo "Building release ${RELEASE} for project ${PROJECT_NAME}..." 

              sh 'chmod +x build.sh' 

              withCredentials([string(credentialsId: 'an-api-key', variable: 'API_KEY')]) { 

                 sh ''' 

                    ./build.sh 

                 ''' 

              } 

           } 

       } 

       stage('Test') { 

           steps { 

              echo "Testing release ${RELEASE}, this is a fake test to mimic failure" 

              script { 

                 if (Math.random() > 0.99) { 

                    throw new Exception() 

                 } 

              } 

           } 

       } 

  } 

  post { 

     success { 

        slackSend channel: '#builds', 

                  color: 'good', 

                  message: "Project ${env.PROJECT_NAME}, success: ${currentBuild.fullDisplayName}." 

     } 

     failure { 

        slackSend channel: '#builds', 

                  color: 'danger', 

                  message: "Project ${env.PROJECT_NAME}, FAILED: ${currentBuild.fullDisplayName}." 

     } 

  } 

} 

 

void auditTools() { 

  sh ''' 

     git version 

     docker version 

  ''' 

} 

and here is the example build.sh 

#!/bin/sh 

python --version 

pip --version 

pip list | grep mlrun 

echo "Using API key: $API_KEY" 

mlrun project -n myproj -u "git://github.com/mlrun/project-demo.git" ./project 

mlrun project -r main -w ./project 

Build Now in Jenkins 

Click on the “Build Now” and wait for a few minutes for the Jenkins build job to complete. (We are using a manual build process for illustration. You can configure the Jenkins project to be triggered by many other ways. Please refer to the Jenkins documentation for more details. You can also create a GitHub webhook to trigger the Jenkins job.) 

After the build is completed, you can check the build logs in Console Output or in the new Jenkins Blueocean UI like below: 

View the MLRun project in Iguazio MLOps Platform 

Once the Jenkins build is complete, you will be able to see the MLRun project in the Iguazio MLOps platform. The MLRun project name is “pipe” here. 

You can see the MLRun job pipeline which starts with condition-1, performs the pre-data step, train the model and test the model.  

The “Monitoring workflow” also shows Overview, Inputs, Artifacts, Results, Logs, Pods of each steps in the workflow 

Wrap-up 

In this article, you learned the steps to: 

1. Run Jenkins in docker 
2. Initially configure Jenkins to run pipeline 
3. Configure the credentials in Jenkins to work with MLRun project 
4. Create a pipeline in Jenkins to deploy a MLRun project in the Iguazio MLOps platform 
5. View the MLRun jobs executed in the MLOps platform  

The key takeaway is that you can kick off an MLRun project from Jenkins, whether it’s a Kubeflow pipeline, or a real-time machine learning model deployment.  

If you have any questions, feel free to get in touch with us on the MLOps Live Slack community.