Vault Meets GitLab-CE: A guide for successful pipeline integration

Picture of Chris Beye

Chris Beye

Systems Architect @ Cisco Systems

Table of Contents

Introduction

Investigating the task of integrating HashiCorp Vault into my existing pipeline, I quickly realized the majority of resources available focused on GitLab-EE and Vault as a SaaS solution. Vault integration with GitLab is a feature that is only available with a premium license. This scenario didn’t align with my specific needs, as I was working with GitLab-CE (Community Edition) on-premises.

The journey was not without its challenges. One of the first hurdles was dealing with certificates 🤮 – a topic I’ve explained in an article. For those without their own Certificate Authority (CA), I recommend revisiting my previous article on using self-signed certificates in GitLab. As we progress, you’ll see why having our own CA file becomes crucial.

I won’t describe the installation details of the Vault server on Ubuntu, as HashiCorp’s documentation covers this comprehensively. Instead, this blog will focus on the unique challenges and solutions encountered while setting up Vault and seamlessly integrating it with an on-prem GitLab-CE pipeline.

The article is based on:
Ubuntu 22.04, Gitlab-CE 16.8.0, Vault 1.15.4

Setting up Hashicorp Vault

Once you have your Vault server started and set up. Keep in mind to note down your root token and seal token! 

Vault automatically generates self-signed certificates which is not what we want to use. As you create your own certificates, change the certificate path in the vault config file /etc/vault.d/vault.hcl

# HTTPS listener
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/198.18.133.99.crt"
  tls_key_file  = "/opt/vault/tls/198.18.133.99.key"
}

Restart the service and navigate to the vault UI. Validate if the right certificates are loaded. 

$ sudo systemctl restart vault.service

Enable jwt authentication, which is the method that is used between GitLab and Vault:

$ sudo vault auth enable jwt
Error enabling jwt auth: Post "https://127.0.0.1:8200/v1/sys/auth/jwt": tls: failed to verify certificate: x509: certificate is valid for 198.18.133.99, not 127.0.0.1

Upps… interesting error message… I will save you some time and directly show you the solution. Keep in mind to use the IP address or DNS name that you set in your “Subject Alternative Name”.

Set the following environment variables:

export VAULT_ADDR='https://198.18.133.99:8200'
export VAULT_CACERT='/opt/vault/tls/ca.crt'
export VAULT_TOKEN='hvs.3me5leR1c8TOgDYTJd'

Now you know why a CA is needed. If you don’t specify the CA you will receive the following error:

x509: certificate signed by unknown authority

Next, the command that creates the auth method to your GitLab server (I installed GitLab and Vault on the same server): 

$ vault write auth/jwt/config jwks_url="https://198.18.133.99/-/jwks"
Create secrets

Log in to the UI, enable a key value (KV) secret engine, and create the first credentials. As shown in the screenshots below:

Of course, you can create the secrets via CLI. Please refer to the Hashicorp documentation.

Vault policy

Create a Vault policy for your secret path, in my case, pod01, which sets the permissions on the secret. I want all content in pod01 to be available with reading capabilities:

$ vault policy write pod01 - <<EOF
path "pod01/*" {
capabilities = [ "read" ]
}
EOF

It took me a while to determine the correct path to use. Some guides suggested using the path starting with secrets/xyz/, which resulted in an error.

Error making API request.
URL: GET https://198.18.133.99:8200/v1/sys/internal/ui/mounts/pod01/dnac
Code: 403. Errors:
* preflight capability check returned 403, please ensure client’s policies grant access to path “pod01/dnac/”

Please keep that in mind.

GitLab project id

Log in to your GitLab and note down the project ID. This information we will use later in the jwt access restriction. 

Vault role access restriction

Create a role access restriction that matches the information that is coming via jws authentication to the policy that we just created, in my case pod01. The highlighted lines need to be changed. In my case, I only will handle requests that are pushed into the main branch.

$ vault write auth/jwt/role/pod01 - <<EOF
{
"role_type": "jwt",
"policies": ["pod01"],
"token_explicit_max_ttl": 60,
"user_claim": "user_login",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "3",
"ref": "main",
"ref_type": "branch"
}
}
EOF
GitLab variables

Create the following 3 variables in GitLab. These are needed for the vault CLI tool to work and the jwt authentification: 

  1. VAULT_AUTH_ROLE = pod01
  2. VAULT_AUTH_PATH = jws
  3. VAULT_ADDR = https://198.18.133.99:8200
Vault container

As I need the ca.crt file to interact with the vault cli, I decided to create my own Docker container and include the ca.crt file.

When using GitLab as a Docker registry, this is the pipeline to build and upload the Dockerfile.

Later, execute the Vault container and provide the secrets previously created.

FROM hashicorp/vault:1.14

RUN apk add jq

COPY ./ca.crt /usr/local/share/ca-certificates/
variables:
  IMAGE_NAME_VAULT: $CI_REGISTRY_IMAGE/vault
  IMAGE_TAG_VAULT: "1.0"  

stages:
  - build
  - vault

build_image:
  stage: build
  tags:
    - shell-runner
  script:
    - docker build -t $IMAGE_NAME_VAULT:$IMAGE_TAG_VAULT docker/vault/.

push_image:
  stage: build
  needs:
    - build_image
  tags:
    - shell-runner
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker push $IMAGE_NAME_VAULT:$IMAGE_TAG_VAULT 

vault:
  stage: vault
  image: $IMAGE_NAME_VAULT:$IMAGE_TAG_VAULT
  tags:
    - docker-runner
  script:
  - export VAULT_CACERT=/usr/local/share/ca-certificates/ca.crt
  - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=$VAULT_AUTH_ROLE jwt=$CI_JOB_JWT)"
  - vault kv get -format json pod01/dnac | jq .data.data | jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]'
Conclusion

Having navigated through the integration of HashiCorp Vault with GitLab-CE, I can attest that while the process may appear straightforward on paper, the reality was far more challenging. 🥵

This experience, however, paves the way for more. My future projects can now leverage Vault more confidently for secure secret storage and management. The successful implementation of Vault into the pipeline marks a significant step in adopting robust secret management practices.

This integration not only aligns with but also reinforces the best practices in security and compliance. It stands as a testament to the importance of a systematic and secure approach in managing sensitive data and credentials in today’s IT environments.

Ressources

More Articles