Deploying Red Hat OpenShift Operators in a disconnected environment

Deploying a Red Hat OpenShift Operator in an environment with internet access is typically straightforward. However, in industries like cyber security or the military sector, where security concerns often prohibit internet access, the process becomes more complex. In a disconnected or air-gapped environment, internet access is usually restricted or unavailable.

In this article, I demonstrate the process of deploying an operator in a disconnected environment. I use the recent Red Hat OpenShift AI operator for this example, because the use of artificial intelligence is becoming crucial to many environments.

Prerequisites

The steps involve saving all required images to a disk, transferring the disk to the disconnected environment, uploading the OpenShift images to a private registry, and then installing the operator

To proceed, you must have the following requirements in place:

• OpenShift is installed in a disconnected environment
• Object storage is available
• A private registry is deployed in the disconnected environment

Mirroring images from public registries to disk

To complete the first task, you use the oc-mirror plugin, an excellent tool for managing and controlling the volume of data being mirrored.

This section requires an internet connection, because you’re fetching images from public registries.

For each operator you want to install in an air-gapped environment, it’s essential to carefully review the documentation. This ensures that you mirror all the necessary images and their dependencies.

The OpenShift AI Operator documentation reveals that several other operators need to be installed first. Similarly, the OpenShift Service Mesh has its own set of dependencies. Missing these dependencies won’t necessarily cause the OpenShift AI Operator installation to fail, but it could lead to issues when using it. For example, if you enable the KServe component when deploying the DataScienceCluster instance, you must also have installed Operators for Red Hat OpenShift Serverless and Red Hat OpenShift Service Mesh in order to support KServe (used by the single-model serving platform to serve large models).

The Openshift AI operator (rhods-operator) has these requirements:

• Pipeline operator
• Serverless operator
• Servicemesh operator
  • Jaeger operator
  • Kiali operator

Red Hat Openshift AI needs extra images that can be found in the Git repository RHOAI disconnected install helper, which includes the list of extra images required for Openshift AI 2.11.

The oc-mirror plugin

Before using the oc-mirror plugin, you need to complete some configuration steps. Fortunately, there is comprehensive documentation available to guide you through this process.

Install the oc-mirror plugin

1. To download the oc-mirror CLI plugin, navigate to the downloads page of the OpenShift Cluster Manager. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file

2. Extract the archive:

   $ tar xvzf oc-mirror.tar.gz
   $ chmod +x oc-mirror

   Do not rename the oc-mirror file

3. To install the oc-mirror CLI plugin, place the file in your PATH. For example:
   $ sudo mv oc-mirror /usr/local/bin/

4. Verify that the plugin for oc-mirror is installed:
   $ oc mirror help

****Configure the oc-mirror credentials****

Next, you must configure the oc-mirror credentials. As usual, read the documentation before attempting this.

1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager

2. Convert your pull secret in JSON format:

   $ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>

3. Copy the JSON file to the .docker directory:

   $ mkdir ~/.docker
   $ cp <path>/<pull_secret_file_in_json> ~/.docker/config.json

****Validate the oc-mirror configuration****

Validate the oc-mirror configuration:

1. Verify that the plugin for oc-mirror is installed:
   $ oc mirror help

   • Note that there is no dash (-) between oc and mirror.
   • If the command returns help options, then oc-mirror is installed.

2. Log in to registry.redhat.io and look for a credentials are valid message:

   $ podman login registry.redhat.io
   Authenticating with existing credentials for registry.redhat.io
   Existing credentials are valid. Already logged in to registry.redhat.io

****Create the oc-mirror ISC (ImageSourceConfiguration) configuration file****

The oc-mirror tool uses an ISC configuration file to define what needs to be mirrored. This configuration allows you to specify the catalog source name and the versions of the operators you want to mirror.

To learn how to automate the creation of ISC files, read my article on developers.redhat.com.

****Find operator channel and release information****

You need to provide the oc-mirror plugin with the operator’s channel and version. To get that information, get a list of the available catalogs:

$ oc mirror list operators --catalogs --version=4.16 Available OpenShift OperatorHub catalogs: OpenShift 4.16: registry.redhat.io/redhat/redhat-operator-index:v4.16 registry.redhat.io/redhat/certified-operator-index:v4.16 registry.redhat.io/redhat/community-operator-index:v4.16 registry.redhat.io/redhat/redhat-marketplace-index:v4.16

Find the available packages within the selected catalog

In this scenario, all operators you’re mirroring are from the Red Hat operator catalogs.

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 NAME DISPLAY NAME DEFAULT CHANNEL 3scale-operator Red Hat Integration - 3scale threescale-2.14 advanced-cluster-management Advanced Cluster Management for Kubernetes release-2.9 amq-broker-rhel8 Red Hat Integration - AMQ Broker for RHEL 8 (Multiarch) 7.11.x amq-online Red Hat Integration - AMQ Online stable amq-streams AMQ Streams stable

Find channels for the selected package

Next, get the channels for the package:

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 --package=rhods-operator NAME DISPLAY NAME DEFAULT CHANNEL rhods-operator Red Hat OpenShift AI fast PACKAGE CHANNEL HEAD rhods-operator alpha rhods-operator.2.11.0 rhods-operator beta rhods-operator.2.4.0 rhods-operator embedded rhods-operator.2.7.0 rhods-operator eus-2.8 rhods-operator.2.8.4 rhods-operator fast rhods-operator.2.11.0 rhods-operator stable rhods-operator.2.10.0 rhods-operator stable-2.10 rhods-operator.2.10.0 rhods-operator stable-2.8 rhods-operator.2.8.4

Find release information in the selected channel

Look in the output for the versions available in the selected channel.

VERSIONS
…
…
2.8.1
1.32.0
2.10.0
2.11.0

Repeat this process for each operator until you’ve gathered this information for each operator you want to mirror.

****Build the oc-mirror ISC configuration file****

If this is the first time you’re creating an ISC file, use the oc mirror command to create a template.

$ oc mirror init > imageset-config.yaml

Edit the new ISC file and update it using the good catalog and the operators release information you gathered in the previous steps. Here’s an example of an ISC configuration file for OpenShift AI 2.11:

$ cat redhat-op-v4.16-config_ai.yaml kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v1alpha2 storageConfig: local: path: /path/to/disk-rh-ai/metadata mirror: operators:

• catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16 targetCatalog: redhat-catalog-v4.16 packages:
  • name: “jaeger-product” channels:
    • name: “stable” minVersion: “1.57.0-7”
  • name: “kiali-ossm” channels:
    • name: “stable” minVersion: “1.73.8”
  • name: “openshift-pipelines-operator-rh” channels:
    • name: “latest” minVersion: “1.15.1”
  • name: “rhods-operator” channels:
    • name: “fast” minVersion: “2.11.0”
  • name: “serverless-operator” channels:
    • name: “stable” minVersion: “1.33.1”
  • name: “servicemeshoperator” channels:
    • name: “stable” minVersion: “2.5.2-0” additionalImages:
  • name: quay.io/integreatly/prometheus-blackbox-exporter@sha256:35b9d2c1002201723b7f7a9f54e9406b2ec4b5b0f73d114f47c70e15956103b5
  • name: quay.io/modh/caikit-nlp@sha256:0cde6c26e02ec398aea959a1a1bcdc615b86821adb41989e81d03de01124545c
  • name: quay.io/modh/caikit-tgis-serving@sha256:4e907ce35a3767f5be2f3175a1854e8d4456c43b78cf3df4305bceabcbf0d6e2 … … …

Finally, create a directory for the image’s TAR file, and then run the command:

$ mkdir disk-rh-ai $ oc mirror --verbose 3 --config=redhat-op-v4.16-config_ai.yaml file://disk-rh-ai

Once complete, the TAR file is in the disk-rh-ai directory you created:

$ ls -ltr disk-rh-ai/ drwxr-xr-x. 3 admin admin 17 Aug 1 10:56 oc-mirror-workspace -rw-r–r--. 1 admin admin 176802807808 Aug 1 11:29 mirror_seq1_000000.tar drwxr-x—. 3 admin admin 21 Aug 1 11:29 metadata

Move the TAR file to the disconnected environment

The operational procedures and security requirements can vary significantly from one organization to another. For example, in certain highly restricted environments, you must copy the TAR file to a portable disk, which you take to your disconnected environment’s security team for review.

Mirroring the operator from disk to private registry

Once the data has been moved to the disconnected environment, you can begin pushing the images in your private registry. Because you’ve used the oc-mirror plugin to push all necessary container images, you must configure the tools accordingly, with the credential information pointing to your private registry instead of the Red Hat public registry.

Here is an example of the oc-mirror credential configuration for a private registry:

$ cat ~/.docker/config.json { "auths": { "quaylab.redhat.com:8443": { "auth": "aW5pdDpVUjiAYzJiOGcxTVc01djQcGh6PQ==", "email": “[email protected]” } }

Upload the container’s images to your private registry

Create a directory to copy the TAR file, and to serve as the destination for “oc mirror” output:

$ mkdir disk-rh-ai $ cp mirror_seq1_000000.tar disk-rh-ai/ $ cd disk-rh-ai

Execute the oc mirror command to upload the container’s images to your private registry

$ oc mirror --verbose 3 \ –from=./mirror_seq1_000000.tar \ docker://quaylab.redhat.com:8443/redhat-v416

Once complete, you get the following directories with similar files:

$ ls -ltr oc-mirror-workspace/results-1722527218/ total 60 drwxr-xr-x. 2 admin admin 6 Aug 1 11:46 charts drwxr-xr-x. 2 admin admin 6 Aug 1 12:47 release-signatures -rw-r–r--. 1 admin admin 50289 Aug 1 12:48 mapping.txt -rwxr-xr-x. 1 admin admin 264 Aug 1 12:48 catalogSource-cs-redhat-catalog-v4-16.yaml -rwxr-xr-x. 1 admin admin 2384 Aug 1 12:48 imageContentSourcePolicy.yaml

Configure the Openshift catalog

Keeping the Openshift default catalogs in a disconnected environment would just trigger unnecessary errors, so disable the default OperatorHub catalog sources:

$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Create a new catalog referring to your private registry using the YAML files generated by the oc-mirror plugin output.

Log in with cluster-admin privileges

You must be logged in to your OpenShift instance as a user with cluster-admin privileges. To confirm:

$ oc login -u admin -p redhat https://api.quaylab.redhat.com:6443 $ oc whoami admin

Create the ICSP file using Openshift CLI

Now you can create the ICSP file:

$ oc apply -f oc-mirror-workspace/results-1722527218/imageContentSourcePolicy.yaml … imagecontentsourcepolicy.operator.openshift.io/generic-0 created imagecontentsourcepolicy.operator.openshift.io/operator-0 created

Create the Red Hat catalog using the Openshift CLI

Next, create the Red Hat catalog:

$ oc apply -f oc-mirror-workspace/results-1722527218/catalogSource-cs-redhat-catalog-v4-16.yaml … catalogsource.operators.coreos.com/cs-redhat-catalog-v4-16 created

Verify that you can see the catalog in the mirrored operators in Operators > OperatorHub:

Configure the OpenShift internal registry

An OpenShift internal registry is a dependency for OpenShift Data Science and OpenShift AI in releases prior to 2.10. For a test cluster, you can use emptyDir. Do not use emptyDir for a production cluster.

$ oc patch configs.imageregistry.operator.openshift.io cluster \ –type merge --patch ‘{"spec":{"storage":{"emptyDir":{}}}}’ \ config.imageregistry.operator.openshift.io/cluster $ oc patch configs.imageregistry.operator.openshift.io cluster \ –type merge --patch ‘{"spec":{"managementState":"Managed"}}’ \ config.imageregistry.operator.openshift.io/cluster

Install the AI operators

I suggest using the OpenShift UI to install the operators. Once you have the OpenShift cluster UI open, follow these steps:

1. Click on Operators > OperatorHub
2. Select the operator to install
3. Keep the default settings and click Install

Repeat this process for each operator:

• Kiali operator
• Jaeger operator
• Pipeline operator
• Serverless operator
• Servicemesh operator
• Openshift AI

Create the RHOAI instance

After the OpenShift AI Operator is successfully installed, you must create a DataScienceCluster instance.

1. In the web console, click Operators > Installed Operators, and then select Red Hat OpenShift AI Operator

2. Click the Data Science Cluster tab

3. Click Create DataScienceCluster

4. In the UI, select YAML view to see the installed components. You have two options:

   • Managed: The Operator actively manages the component, installs it, and tries to keep it active. The Operator upgrades the component only if it is safe to do so
   • Removed: The Operator actively manages the component but does not install it. If the component is already installed, the Operator tries to remove it

5. Click the Create button to continue

Access the Red Hat AI dashboard

At this point,the Openshift AI Data Science cluster instance has been successfully deployed. The OpenShift AI dashboard has its own UI.

1. To access the OpenShift AI dashboard, click the Grid icon application launcher at the top of the screen

2. Right-click on Red Hat OpenShift AI and copy the URL of your OpenShift AI instance

3. Paste the URL into a web browser to navigate to the dashboard

Conclusion

Working in a disconnected environment presents significant challenges and adds complexity to the application development process. This mode of operation requires enhanced control over how your applications interact with the internet. Using the right tools is crucial for managing data efficiently and minimizing your workload. In this blog, I’ve demonstrated how to use the oc-mirror plugin, which I believe is the best approach for this scenario. It provides enhanced control over the volume of data you need to handle, streamlines the process, and improves efficiency.

If you’re interested in knowing more about Red Hat AI, click here. If you want to know more about Openshift in disconnected environment, click here.