Openshift Update Service disconnected

The OpenShift Update Service (OSUS) builds a graph of update possibilities based on release images in the registry. The graph is based on recommended, tested update paths from a specific version. This info comes from Internet so to be able to get this service offline we need to do some changes:

First of all, we need a to install Openshift Update Service Operator following next steps:

Create a yaml file with the namespace, the operator group and de subscription (10-update-service-operator.yaml)

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-update-service
  annotations:
    openshift.io/node-selector: ""
  labels:
    openshift.io/cluster-monitoring: "true"
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: update-service-operator-group
  namespace: openshift-update-service
spec:
  targetNamespaces:
  - openshift-update-service
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: update-service-subscription
  namespace: openshift-update-service
spec:
  channel: v1
  installPlanApproval: "Automatic"
  source: "redhat-operator-index"
  sourceNamespace: "openshift-marketplace"
  name: "cincinnati-operator"

Apply it

oc apply -f 10-update-service-operator.yaml

It results:

namespace/openshift-update-service created
operatorgroup.operators.coreos.com/update-service-operator-group created
subscription.operators.coreos.com/update-service-subscription created

Configure registry certificate to trusted store. Create a file with the configmap (10-my-registry-ca.yaml). Substitute accordingly with the certificate in the ca.crt from the home directory.

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-ca-registry
  namespace: openshift-config
data:
  updateservice-registry: |
    -----BEGIN CERTIFICATE-----
    MIIFKzCCAxOgAwIBAgIUOwRTWpPLn6/4nMepKoivyBNY2IIwDQYJKoZIhvcNAQEL
    ...
    HQTLljb0wWdsLQ8juGIGMA7WmKhoTaYXcs2lJ8W1C9ioZJMVJOtQCrVmJrfc25PR
    jet5zqdfRDxalVS0nOI/qakkKoYnhoTyZuFuxicRAA==
    -----END CERTIFICATE-----
  registry.dsal..8443: |
    -----BEGIN CERTIFICATE-----
    MIIFKzCCAxOgAwIBAgIUOwRTWpPLn6/4nMepKoivyBNY2IIwDQYJKoZIhvcNAQEL
    ...
    HQTLljb0wWdsLQ8juGIGMA7WmKhoTaYXcs2lJ8W1C9ioZJMVJOtQCrVmJrfc25PR
    jet5zqdfRDxalVS0nOI/qakkKoYnhoTyZuFuxicRAA==
    -----END CERTIFICATE-----

And apply it:
```
oc apply -f 10-my-registry-ca.yaml
```

enable CA in proxy also (user-ca-bundle)

oc patch image.config.openshift.io/cluster --type=merge --patch='{"spec":{"additionalTrustedCA:":{"name":"custom-ca-registry"}}}'

Create graph update image (10-update-service-dockerfile)

FROM registry.access.redhat.com/ubi8/ubi:8.1

RUN curl -L -o cincinnati-graph-data.tar.gz https://github.com/openshift/cincinnati-graph-data/archive/master.tar.gz

CMD exec /bin/bash -c "tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati/graph-data/ --strip-components=1"

Build and push:

podman build -f ./10-update-service-dockerfile -t registry.dsal:8443/openshift/graph-data:latest

podman push --authfile pull-secret-all.json registry.dsal:8443/openshift/graph-data:latest

Another approach to generating an OUS-graph image is by including the graph:true option in the imageset config, which enables oc-mirror to create it. Something like this:

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
storageConfig:
  registry:
    imageURL: registry.dsal:8443/imagetset/mirror-disconnected:metadata
    skipTLS: true
mirror:
  platform:
    channels:
    - name: stable-4.11
      type: ocp
      minVersion: 4.11.20
      shortestPath: true
    graph: true (1)
    architectures:
    - amd64

1	This options will create an graph image for OUS. In our examples would be: `registry.dsal:8443/mirror/openshift/graph-image:latest`

Now we will create an instance of the UpdateService object (CR from the Update Service Operator)(10-update-service-UpdateService.yaml)

apiVersion: updateservice.operator.openshift.io/v1
kind: UpdateService
metadata:
  name: service
  namespace: openshift-update-service
spec:
  replicas: 2
  graphDataImage: registry.dsal:8443/openshift/graph-data:latest
  releases: registry.dsal:8443/mirror/openshift/release-images

and apply it:

oc apply -f 10-update-service-UpdateService.yaml

Now, update clusterVersion with graph url (we will use the fqdn from the k8s service dns)

POLICY_ENGINE_GRAPH_URI="http://service-policy-engine.openshift-update-service/api/upgrades_info/v1/graph"

oc patch clusterversion version -p "{\"spec\":{\"upstream\":\"${POLICY_ENGINE_GRAPH_URI}\"}}" --type merge

Alternatively, if you prefer to use the public route, you can execute the following command:

POLICY_ENGINE_GRAPH_URI="$(oc -n "openshift-update-service" get -o jsonpath='{.status.policyEngineURI}/api/upgrades_info/v1/graph{"\n"}' updateservice "service")"

oc patch clusterversion version -p "{\"spec\":{\"upstream\":\"${POLICY_ENGINE_GRAPH_URI}\"}}" --type merge

Now, check everything is working:

$ oc get pod -n openshift-update-service
NAME                                     READY   STATUS    RESTARTS   AGE
service-86ff6b6df-82fn4                  2/2     Running   0          51m
service-86ff6b6df-9fc7m                  2/2     Running   0          51m
updateservice-operator-ccb64988d-f97v6   1/1     Running   0          17h

$ oc adm upgrade
Cluster version is 4.11.20

Upstream: http://service-policy-engine.openshift-update-service/api/upgrades_info/v1/graph
Channel: stable-4.11 (available channels: candidate-4.11, candidate-4.12, fast-4.11, fast-4.12, stable-4.11)
No updates available. You may force an upgrade to a specific release image, but doing so may not be supported and may result in downtime or data loss.

As you can see, there isn’t any version to upgrade. This is because our update service takes a look to the registry to discover releases. Great, isn’t it?

We are going to mirror a new Openshift release. Create a new imageset for oc-mirror (10-update-service-imageset.yaml)

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
archiveSize: 1
storageConfig:
  registry:
    imageURL: registry.dsal:8443/imagetset/mirror-disconnected:metadata
    skipTLS: true
mirror:
  platform:
    channels:
    - name: stable-4.11
      type: ocp
      minVersion: 4.11.20
      shortestPath: true
    architectures:
    - amd64

Mirror and apply and add/update (just in case) the ImageContentSourceFile:

oc-mirror --config=./10-update-service-imageset.yaml  docker://registry.dsal:8443/mirror

Apply results folder:

oc apply -f oc-mirror-workspace/results-xxxxx/

Now, if we execute oc adm upgrade we will see new version to upgrade:

Cluster version is 4.11.20

Upgradeable=False

  Reason: InsightsNotUpgradeable
  Message: Cluster operator insights should not be upgraded between minor versions: Unable to report: unable to build request to connect to Insights server: Post "https://console.redhat.com/api/ingress/v1/upload": dial tcp 213.57.23.152:443: i/o timeout

Upstream: http://service-policy-engine.openshift-update-service/api/upgrades_info/v1/graph
Channel: stable-4.11 (available channels: candidate-4.11, candidate-4.12, fast-4.11, fast-4.12, stable-4.11)

Recommended updates:

  VERSION     IMAGE
  4.11.31     registry.dsal:8443/mirror/openshift/release-images@sha256:5fbe52f0f89d72e4d28b2a40dc69174fe10cce0a99dc5caa6fcfbf4226e08919
  4.11.30     registry.dsal:8443/mirror/openshift/release-images@sha256:8230ca19fea80ef02f255a9f92688aa2639f68739a2b69114bf9af06080f9edc
  4.11.29     registry.dsal:8443/mirror/openshift/release-images@sha256:1105aa27f627a99a2b3a8b6257a12697b2033a44f1fa2af41491a8e66cd279ac
  4.11.28     registry.dsal:8443/mirror/openshift/release-images@sha256:85238bc3eddb88e958535597dbe8ec6f2aa88aa1713c2e1ee7faf88d1fefdac0
  4.11.27     registry.dsal:8443/mirror/openshift/release-images@sha256:65e71a774a18c1c191f28655ce245abeecd653e8215b75f87eb23ceadacd530d
  4.11.26     registry.dsal:8443/mirror/openshift/release-images@sha256:1c3913a65b0a10b4a0650f54e545fe928360a94767acea64c0bd10faa52c945a
  4.11.25     registry.dsal:8443/mirror/openshift/release-images@sha256:2adcf72e10e67ace02ade32467ff7e75680ec1c71545a038196e569dc3149ad0
  4.11.24     registry.dsal:8443/mirror/openshift/release-images@sha256:36ee0fd41073248dc566350db67bd52d2bed6e1691ab11879379b462d740e721
  4.11.22     registry.dsal:8443/mirror/openshift/release-images@sha256:6e16fe4b05385d8422529c0120aff73f55a55eff57581a0714443d92a87f1ce9
  4.11.21     registry.dsal:8443/mirror/openshift/release-images@sha256:860cc37824074671c4cf76e02d224d243e670d2298e6dab8923ee391fbd0ae1c