Skip to content

orbit-setup.md

Latest commit

 

History

History
331 lines (234 loc) · 10.9 KB

orbit-setup.md

File metadata and controls

331 lines (234 loc) · 10.9 KB
Raw
stage Analytics
group Context Systems
info To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
description Install GitLab Orbit on Kubernetes, connect GitLab to it, and index your first group.
title Set up GitLab Orbit

{{< details >}}

  • Tier: Premium, Ultimate
  • Offering: GitLab Self-Managed

{{< /details >}}

GitLab Orbit runs as a Helm release next to your GitLab instance.

Prerequisites:

  • Data replication running, with rows arriving in the data lake.
  • A ClickHouse administrator account.
  • Administrator access to GitLab.

Set up GitLab Orbit in this order:

  1. Turn on GitLab Orbit in GitLab.
  2. Create the Secrets.
  3. Install GitLab Orbit.
  4. Turn on indexing for a group.

Turn on GitLab Orbit in GitLab

GitLab and GitLab Orbit authenticate with one shared key, and GitLab owns it. Turn on GitLab Orbit in GitLab first, so that the key exists before you copy it to the cluster.

GitLab connects to the gRPC endpoint on port 50054, and an endpoint that begins with tls:// uses TLS. To use a service mesh for encryption instead, see Use a plaintext connection.

{{< tabs >}}

{{< tab title="Linux package (Omnibus)" >}}

  1. Edit /etc/gitlab/gitlab.rb:

    gitlab_rails['orbit_enabled'] = true
gitlab_rails['orbit_grpc_endpoint'] = 'tls://orbit.example.com:50054'

  2. Reconfigure GitLab:

    sudo gitlab-ctl reconfigure

  3. Copy the key that GitLab generated. Do not re-encode or trim it:

    sudo cat /var/opt/gitlab/gitlab-rails/etc/gitlab_knowledge_graph_secret

Warning

If you run several Rails nodes, or a tool such as Ansible replaces /etc/gitlab/gitlab-secrets.json, set gitlab_rails['orbit_secret'] to the key in gitlab.rb. Otherwise, a reconfigure can generate a new key, and GitLab Orbit rejects every request.

{{< /tab >}}

{{< tab title="Helm chart (Kubernetes)" >}}

The GitLab chart does not generate the key. Create the Secret before you upgrade GitLab, or new GitLab pods stay in ContainerCreating.

  1. Generate a key:

    openssl rand -base64 32

  2. Store the key in a Secret in the GitLab namespace:

    kubectl -n <gitlab_namespace> create secret generic gitlab-orbit-jwt --from-literal=secret='<key>'

  3. Reference the Secret in the GitLab chart values:

    global:
  appConfig:
    orbit:
      enabled: true
      grpcEndpoint: 'tls://orbit.example.com:50054'
      jwtSecret:
        secret: gitlab-orbit-jwt
        key: secret

{{< /tab >}}

{{< /tabs >}}

GitLab logs connection errors until you install GitLab Orbit.

Create the Secrets

GitLab Orbit reads the GitLab key and the ClickHouse passwords from a Kubernetes Secret. The chart setup hook uses the administrator password to create the GitLab Orbit users.

To create the Secrets:

  1. Create the namespace:

    kubectl create namespace gitlab-orbit

  2. Create the Secret:

    kubectl -n gitlab-orbit create secret generic orbit-secrets \
  --from-literal=gitlab-jwt-verifying-key='<gitlab_key>' \
  --from-literal=gitlab-jwt-signing-key='<gitlab_key>' \
  --from-literal=admin-password='<clickhouse_admin_password>' \
  --from-literal=datalake-password='<siphon_reader_password>' \
  --from-literal=graph-password='<writer_password>' \
  --from-literal=graph-read-password='<reader_password>'

    Replace the following values:

    • <gitlab_key>: The key from Turn on GitLab Orbit in GitLab.
    • <clickhouse_admin_password>: The password of the ClickHouse administrator account.
    • <siphon_reader_password>, <writer_password>, and <reader_password>: New passwords for the gkg_siphon_reader, gkg_writer, and gkg_reader users.

  3. If the webserver terminates TLS, create a TLS Secret for the gRPC endpoint:

    kubectl -n gitlab-orbit create secret tls orbit-webserver-tls --cert=<cert_file> --key=<key_file>

In production, sync the Secrets from your secret manager, for example with the External Secrets Operator. The pods read the Secrets when they start, so restart them after a change:

kubectl -n gitlab-orbit rollout restart deployment

Install GitLab Orbit

To install GitLab Orbit:

  1. Save the following as orbit-values.yaml:

    secrets:
  gitlabJwtVerifyingKey: {secretName: orbit-secrets}
  gitlabJwtSigningKey: {secretName: orbit-secrets}
  datalakePassword: {secretName: orbit-secrets}
  graphPassword: {secretName: orbit-secrets}
  graphReadPassword: {secretName: orbit-secrets}

clickhouseSetup:
  enabled: true
  adminSecret:
    secretName: orbit-secrets

clickhouse:
  datalake:
    host: <clickhouse_host>
    database: gitlab_clickhouse_main_production
    user: gkg_siphon_reader
  graph:
    host: <clickhouse_host>
    database: orbit
    user: gkg_writer
    readUser: gkg_reader

dispatcher:
  enabled: true

nats:
  url: nats://nats.nats.svc.cluster.local:4222

gitlab:
  # The GitLab address the cluster reaches, such as an internal address or load balancer.
  # If it differs from the certificate host name, set resolveHost. See Networking.
  baseUrl: https://gitlab.example.com
  # resolveHost: gitlab.internal

tls:
  enabled: true
  existingSecret: orbit-webserver-tls

    Replace <clickhouse_host> with the host name of your ClickHouse server.

  2. Install the chart:

    helm upgrade --install orbit oci://registry.gitlab.com/gitlab-org/orbit/orbit-helm-charts/orbit \
  --namespace gitlab-orbit --values orbit-values.yaml

  3. Confirm that the webserver, indexer, and dispatcher pods are Running:

    kubectl -n gitlab-orbit get pods

Adjust the values for your environment:

  • For ClickHouse over TLS, such as ClickHouse Cloud, add httpPort: 8443 and ssl: true to both ClickHouse connections.
  • If the ClickHouse certificate comes from your own CA, add the CA certificate to trust.extraCAs.
  • If a load balancer or a service mesh handles TLS, remove the tls block.
  • If NATS requires a password, set nats.username, add nats-password to the Secret and natsPassword to secrets, and turn on nats.tls so the password does not cross the network in clear text.
  • Snowplow analytics is off by default. To turn it on, set analytics.enabled: true.

For TLS, private certificate authorities, NATS TLS, and plaintext connections, see Networking for GitLab Orbit. For every value and its default, see the GitLab Orbit Helm chart. For pod sizes, see GitLab Orbit sizing.

Billing requirements

GitLab Orbit reports usage to GitLab and checks that your subscription has GitLab Credits available. The Helm chart configures billing by default. Allow outbound HTTPS from the webserver to customers.gitlab.com and billing.prdsub.gitlab.net. For more information, see Billing connections.

When your subscription has no GitLab Credits available, queries fail with the GitLab credits exhausted error.

Turn on indexing for a group

GitLab Orbit indexes top-level groups, with their subgroups and projects.

To turn on indexing for a group:

  1. In the upper-right corner, select Admin.
  2. Go to the GitLab Orbit configuration page at /admin/orbit.
  3. Under Top-level groups, find your group and select Turn on GitLab Orbit.
  4. Confirm.

To index every top-level group, turn on Turn on GitLab Orbit for all top-level groups automatically.

You can also turn on indexing with the API. Use a personal access token of an administrator, with the api scope. If Admin Mode is on, the token also needs the admin_mode scope.

curl --request PUT \
  --header "PRIVATE-TOKEN: <your_access_token>" \
  --url "https://gitlab.example.com/api/v4/admin/knowledge_graph/namespaces/<group_id>"

Replace <group_id> with the ID or URL-encoded path of a top-level group. For a subgroup, the API returns 400 Bad Request.

The first index takes from several minutes to a few hours. An hourly sweep picks up any data that indexing missed.

Verify the installation

Query the graph for the projects in the group that you turned on. The query uses GQL, which the supported GitLab versions accept by default.

To verify the installation:

  1. Save the query as query.gql:

    MATCH (p:Project) WHERE p.full_path STARTS WITH 'your-group/' RETURN p.name, p.full_path LIMIT 10

    Replace your-group/ with the full path of your group, followed by /.

  2. Send the query:

    jq -Rs '{query: ., response_format: "raw"}' query.gql | curl --request POST \
  --header "Authorization: Bearer <your_access_token>" \
  --header "Content-Type: application/json" \
  --data @- \
  --url "https://gitlab.example.com/api/v4/orbit/query"

    Replace <your_access_token> with your personal access token.

The query returns rows after the first index finishes. Pod status and the /-/readiness endpoint pass before that point, so they do not confirm indexing.

Upgrade GitLab Orbit

An upgrade reuses the graph database. When a release changes the schema, GitLab Orbit builds the new version next to the active one, and switches after indexing finishes. Plan ClickHouse disk for a second copy of the graph.

To upgrade GitLab Orbit:

  1. Confirm that the new GitLab Orbit version supports your GitLab, Helm chart, and ClickHouse versions. See Version requirements.
  2. Run the helm upgrade command from Install GitLab Orbit.
  3. Run the query from Verify the installation.

Upgrade GitLab

When you upgrade GitLab, upgrade Siphon and GitLab Orbit with it.

To upgrade GitLab:

  1. Confirm that your GitLab Orbit version supports the new GitLab version. See Version requirements.

  2. Upgrade GitLab.

  3. In siphon-values.yaml, set global.gitlabVersion to the new GitLab version, then upgrade Siphon.

  4. Upgrade GitLab Orbit.

  5. On the Linux package, confirm that the key in GitLab matches the key in the Secret:

    sudo cat /var/opt/gitlab/gitlab-rails/etc/gitlab_knowledge_graph_secret
    kubectl -n gitlab-orbit get secret orbit-secrets -o jsonpath='{.data.gitlab-jwt-verifying-key}' | base64 -d

Related topics