Insights and Telemetry

On an OpenShift Cluster, there are certain services, such as Insight or Telemetry, which require Internet connectivity to function properly. These services continuously send information to the Red Hat Cloud Portal. However, as our cluster is currently disconnected from the Internet, we need to find a workaround to upload this information in a controlled manner. Therefore, we have several options to resolve this disconnected situation:

Disable both services
Manual upload (jobs sending info via proxy)
Using a cluster-wide proxy server

Let’s see how to do it in both services. We will start with Insight Service.

Insights

Disable Insights

Download the global cluster pull secret to your local file system:
```
oc extract secret/pull-secret -n openshift-config --to=.
```

Edit file .dockerconfigjson (make a copy before), and remove the cloud.openshift.com JSON entry.

Take in account that in our disconnected cluster we have the global secret configured only with the credentials of the private registry. So, in this case, we don’t have to modify anything. :)

Update global secret:

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=.dockerconfigjson

Manual upload (job via proxy)

For this option, we will generate the info/data and then upload it to the Red Hat Cloud services. However, in order to do so, we require a proxy server or another machine with Internet access. In our specific case, we will create and utilize a proxy. To accomplish this, we will run a container within the Registry VM that will host the proxy server.

Run a proxy in the registry VM ( access to the registry VM using kcli ssh registry` and become root sudo su before)
```
podman run -d --name squid-container -e TZ=UTC -p 3128:3128 docker.io/ubuntu/squid:latest
```
Exit from Registry VM and stay in the Hypervisor.

Extract token from pull secret for cloud.redhat.com.

cat ~/pull-secret-all.json | jq '.auths["cloud.openshift.com"].auth'

Extract clusterid from cluster:

oc get clusterversion -o jsonpath='{.items[].spec.clusterID}{"\n"}'

Create job and update environment variables TOKEN, CLUSTERID and HTTPS_PROXY properly with the above values (8-insights-cronjob.yaml)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: insights-operator-cronjob
  namespace: openshift-insights
  annotations:
    config.openshift.io/inject-proxy: insights-operator
spec:
  schedule: "*/30 * * * *"          
  timeZone: Etc/UTC                
  concurrencyPolicy: "Replace"     
  startingDeadlineSeconds: 200     
  suspend: false                    
  successfulJobsHistoryLimit: 3    
  failedJobsHistoryLimit: 1        
  jobTemplate:                     
    spec:
      template:
        metadata:
          name: insights-operator-cronjob
          namespace: openshift-insights
          annotations:
            config.openshift.io/inject-proxy: insights-operator  
        spec:
          restartPolicy: OnFailure
          serviceAccountName: operator
          nodeSelector:
            kubernetes.io/os: linux
            node-role.kubernetes.io/master: ""
          tolerations:
          - effect: NoSchedule
            key: node-role.kubernetes.io/master
            operator: Exists
          - effect: NoExecute
            key: node.kubernetes.io/unreachable
            operator: Exists
            tolerationSeconds: 900
          - effect: NoExecute
            key: node.kubernetes.io/not-ready
            operator: Exists
            tolerationSeconds: 900
          volumes:
          - name: snapshots
            emptyDir: {}
          - name: service-ca-bundle
            configMap:
              name: service-ca-bundle
              optional: true
          initContainers:
          - name: gather-data
            image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78e5a3b2244793c6d11b05332244ab5598b0b8c0332dd39b41a42913afc517e0
            terminationMessagePolicy: FallbackToLogsOnError
            volumeMounts:
            - name: snapshots
              mountPath: /var/lib/insights-operator
            - name: service-ca-bundle
              mountPath: /var/run/configmaps/service-ca-bundle
              readOnly: true
            ports:
            - containerPort: 8443
              name: https
            resources:
              requests:
                cpu: 10m
                memory: 70Mi
            args:
            - gather
            - -v=4
            - --config=/etc/insights-operator/server.yaml
          containers:
            - name: uploader
              image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0671b1b4e197c7a2673f9fbee100e5c1d76a49318fe436cabf2809279f4f5ce4
              env:
              - name: TOKEN
                value: "fasdfasdfasdfasdfas"
              - name: CLUSTERID
                value: "xxxx-xxxx-xxxx-xxxx"
              - name: HTTPS_PROXY
                value: "registry.dsal:3128"
              command:
              - /bin/sh
              args:
                - -c
                - 'sleep 30s && ls -d /var/lib/insights-operator/*gz && curl -v -H "User-Agent: insights-operator/one10time200gather184a34f6a168926d93c330 cluster/$(CLUSTERID)" -H "Authorization: Bearer $(TOKEN)" -F "upload=@$(ls -d /var/lib/insights-operator/*gz | head -n 1); type=application/vnd.redhat.openshift.periodic+tar" https://console.redhat.com/api/ingress/v1/upload'
              volumeMounts: [{name: snapshots, mountPath: /var/lib/insights-operator}]

and "install" the cronjob:

oc apply -f 8-insights-cronjob.yaml

This is a cronjob running every 30 minutes. You can change the schedule expression to run again the job. Alternatively, you can create a job based on this cronjob executing:

oc create job -n openshift-insights --from=cronjob.batch/insights-operator-cronjob test

Telemetry

The telemetry service/client is a component of Openshift Monitoring. For this service there are also the same 3 options:

Disable: Same way as Insight, removing credentials from global pull-secret. (We won’t cover this method)
Manual upload (job via proxy)
Using a cluster-wide proxy

Manual Upload

Actually, it’s an "unsupported" way of forcing sending metrics data.

This service is only installed if in the global pull-secret are the credentials for cloud.redhat.com. So, if in your cluster is not configured, added it again.

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=./pull-secret-all.json

Once added, wait some minutes, or force deletion of the operator pod to reconcile, and it can (re)create the telemeter-client pod.

oc delete pod -l app=cluster-monitoring-operator -n openshift-monitoring

Wait until telemeter pod is created:

$ oc get pod -n openshift-monitoring | grep telemeter-client
telemeter-client-85d9d5fcbc-49chl                        3/3     Running   0          16s

To enable the telemeter pod to use the proxy server for sending data, we simply need to add the HTTPS_PROXY variable to the deployment. This can be accomplished by executing the following command:

oc set env -n openshift-monitoring deployment/telemeter-client -c telemeter-client HTTPS_PROXY=registry.dsal:3128

When Monitoring Operator reconciles the changes will be lost, so we can execute this command in a k8s cronjob or externally on demand.

To check the status of insight and telemetry, browse in https://console.redhat.com/openshift