Towards a CI/CD world without credentials

Temps de lecture : 8 minutes

Did you know about the largest database account leaked ? More than 12 millions account credentials have been leaked.

And beside that, in our traditional CI/CD platforms, we store credentials for cloud deployments (access key/secret key, service account, login/password) somewhere to be in the end available as an environment variable. But sometimes GitLab or GitHub had also some exposure.

Great, what are the other problems ? Here are some:

  • Security breach on the CI/CD platform
  • Password copy/paste on your desktop computer
  • Rotating secrets every 3 months
  • Ever lasting credentials available for every one with a simple echo

Not really fun.

So today i’m writing this article to help you to change one of a common bad habit and work with Cloud without saving any credentials in your CI/CD. To do so we’ll setup an identity provider and generate temporary credentials to limit the risk as much as we can in case of exposure.

Maybe you already read our post on GitLab and Vault, it is similar but we will go a step further and remove the need of a vault.

Prerequisites

For this article you will need to execute commands to setup your cloud platform. So you will need theses:

  • AWS
  • GCP

Setup your Cloud

Getting to the point, you will need to setup your favorite Cloud platform to trust your favorite CI/CD. Be smart and pick the only ones regarding your needs.

The identity provider of CI/CD platform is a key element of this mechanism. Because we trust it, it must guaranty that all signed identity must be valid. The other element is the identity federation in your cloud. It is the one that will guaranty the chain of trust.

Setup AWS

1- Create the Identity federation

The IAM Identity provider is a resource in AWS that help you assume a role by trusting an external identity provider. In our case we will use the Open ID Connect identity provider standard.

# If you're using Gitlab
IDENTITY_PROVIDER_HOST='gitlab.com'
# If you're using GitHub
IDENTITY_PROVIDER_HOST='token.actions.githubusercontent.com'

# Trust the Identity Provider
fingerprint="$(openssl s_client -connect "${IDENTITY_PROVIDER_HOST}:443" </dev/null 2>/dev/null |
    openssl x509 -fingerprint -noout | awk -F= '{ print $2 }' |
    sed 's/://g')"
aws iam create-open-id-connect-provider \
    --url "https://${IDENTITY_PROVIDER_HOST}" --thumbprint-list "${fingerprint}" \
    --client-id-list "https://${IDENTITY_PROVIDER_HOST}"

2- Create a CI/CD role

To have an access level in AWS, all come with permission policies. These permissions can be attached to a user, and in a federation scenario as currently, we can attach it to a role. In AWS, a role is an entity having temporary credentials tied to a set of permissions in the AWS cloud. The role is used to perform all your AWS cloud actions. And because we trust our created Identity Provider, we allow it to assume a role only if the request is coming from the right GitLab project or GitHub repository.

Note: The IAM Identity provider and the role must be in the same account. 

AWS_ACCOUNT_ID='1234567890'

# Role for a specific GitLab project
GITLAB_PROJECT_PATH='your-group/your-project'

aws iam create-role --role-name 'cicd' --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Principal": {
                "Federated": "'"arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${IDENTITY_PROVIDER_HOST}"'"
            },
            "Condition": {
                "StringLike": {
                    "'"${IDENTITY_PROVIDER_HOST}:sub"'": "'"project_path:${GITLAB_PROJECT_PATH}:ref_type:*:ref:*"'"
                }
            }
        }
    ]
}'

# Role for specific GitHub repository
GITHUB_REPOSITORY_PATH='your-org/your-repository'

aws iam create-role --role-name 'cicd' --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Principal": {
                "Federated": "'"arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${IDENTITY_PROVIDER_HOST}"'"
            },
            "Condition": {
                "StringLike": {
                    "'"${IDENTITY_PROVIDER_HOST}:sub"'": "'"repo:${GITHUB_REPOSITORY_PATH}:*"'"
                }
            }
        }
    ]
}'

3- Attach policy to your role

To be able to perform actions with your role, you should attach policies. Let’s see an example with S3 read only actions. Be careful and respect the Least Privilege Principle.

aws iam attach-role-policy --role-name 'cicd' --policy-arn 'arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess'

You can repeat this command as much as you need to attach all necessaries policies to allow your CI/CD to deploy your infrastructure and application.

Setup GCP

1- Create the Identity Federation

The Workload Identity Federation is a resource used to trust an external identity provider (Here GitLab or GitHub) and then once the Identity has been confirmed, try to assume a targeted service account. 

export GCP_WIF_PROJECT_ID='your-project-id-for-workload-identity-federation'

# Create the pool
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools create --location global cicd

# Trust the GitLab Identity Provider
export GITLAB_ORGANIZATION='your-organization'
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools providers create-oidc \
    --location=global --workload-identity-pool=cicd gitlab \
    --issuer-uri='https://gitlab.com' --allowed-audiences='https://gitlab.com' \
    --attribute-mapping='google.subject="gitlab::" + assertion.project_id,attribute.root_group=assertion.sub.extract("project_path:{root_group}/"),attribute.project_id=assertion.project_id' \
    --attribute-condition='attribute.root_group == "'"${GITLAB_ORGANIZATION}"'"'

# Trust the GitHub Identity Provider
export GITHUB_ORGANIZATION='your-organization'
gcloud --project="${GCP_WIF_PROJECT_ID}" iam workload-identity-pools providers create-oidc \
    --location=global --workload-identity-pool=cicd gitlab \
    --issuer-uri='https://token.actions.githubusercontent.com' \
    --attribute-mapping='google.subject="github::" + assertion.repository_id,attribute.repository_owner=assertion.repository_owner,attribute.repository_id=assertion.repository_id' \
    --attribute-condition='attribute.repository_owner == "'"${GITHUB_ORGANIZATION}"'"'

2- Create your service account

To have an access level in GCP, all come with permissions. These permissions can be attached to a user, and in a federation scenario as currently, we can attach it to a service account. In GCP, a service account is an identity having temporary credentials tied to a set of permissions in the GCP cloud.

The service account is used to perform all your google cloud actions. And because we trust our created Workload Identity Pool, we allow it to impersonate a service account only if the request is coming from the right GitLab project or GitHub repository.

export GCP_PROJECT_ID='your-project-id'
export GCP_WIF_PROJECT_ID='your-project-id-for-workload-identity-federation'
export GCP_WIF_PROJECT_NUMBER=$(gcloud projects describe "${GCP_WIF_PROJECT_ID}" --format=value(projectNumber))

# Create your service account
gcloud --project="${GCP_PROJECT_ID}" iam service-accounts create cicd

# Allow impersonation for the gitlab project
gcloud --project="${GCP_PROJECT_ID}" iam service-accounts add-iam-policy-binding "cicd@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/iam.workloadIdentityUser" \
    --member="principalSet://iam.googleapis.com/projects/${GCP_WIF_PROJECT_NUMBER}/locations/global/workloadIdentityPools/cicd/attribute.project_id/${GITLAB_PROJECT_ID}"

# Allow impersonation for the github repository
gcloud iam service-accounts add-iam-policy-binding cicd \
    --role="roles/iam.workloadIdentityUser" \
    --member="principalSet://iam.googleapis.com/projects/${GCP_WIF_PROJECT_NUMBER}/locations/global/workloadIdentityPools/cicd/attribute.repository_id/${GITHUB_REPOSITORY_ID}"

Note: The Workload Identity Federation and the service account can be in separated projects.

Setup your favorite CI/CD platform

Now that we have setup the trust chain, we will just need to ask for temporary credentials. Lets wrap this into a cloud_authentication.sh file to be more readable and reusable.

Let’s begin with a schema to explain how it is working.

  1. Your trusted identity provider will generate a signed token to identify the job and where and when it is running
  2. The cloud_authenticate.sh above will use this signed token to ask for temporary credentials
  3. During the ask for credential, the cloud identity federation will verify the signature of the token to ensure it is valid
  4. Finally the cloud identity federation generate temporary credentials to send then back to the cloud_authenticate.sh

Now let’s do it…

Setup your GitLab job

Your-Awesome-Job:
  script:
    - source cloud_authentication.sh
    - echo "your cloud stuff"

Setup your GitHub action job

jobs:
  Your-Awesome-Job:
    permissions:
      id-token: write
    steps:
    - run: source cloud_authentication.sh
    - run: echo "your cloud stuff"

Implement your script for AWS

#!/usr/bin/env bash
# @file cloud_authentication.sh

export AWS_ACCOUNT_ID='00000000' # <= replace with your AWS account id where your role and IAM identity provider is setup

aws_assume_role() {
    local credentials
    mapfile -t credentials < <(aws_sts_assume_role_with_web_identity "$@" | aws_sts_to_credentials)
    if [[ -n "${credentials+x}" ]]; then
        export AWS_ACCESS_KEY_ID=${credentials[0]}
        export AWS_SECRET_ACCESS_KEY=${credentials[1]}
        export AWS_SESSION_TOKEN=${credentials[2]}
        true
    else
        false
    fi
}

aws_sts_assume_role_with_web_identity() {
    local account_id=$1
    local role_name=$2

    aws sts assume-role-with-web-identity \
        --role-arn "arn:aws:iam::${account_id}:role/${role_name}" \
        --role-session-name "$(role_session_name)" \
        --web-identity-token "$(jwt_token)" \
        --duration-seconds 3600
}

role_session_name() {
    if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
        echo "GitLabProject-${CI_PROJECT_ID}-Job-${CI_JOB_ID}"
    elif [[ -n "${CI+x}" ]]; then # GitHub
        echo "GitHubRepository-${GITHUB_REPOSITORY_ID}-Job-${GITHUB_JOB}"
    else
        exit 2
    fi
}

jwt_token() {
    if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
        echo "${CI_JOB_JWT_V2}"
    elif [[ -n "${CI+x}" ]]; then # GitHub
        echo "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
    else
        exit 2
    fi
}

aws_sts_to_credentials() {
    jq -r '.Credentials | .AccessKeyId,.SecretAccessKey,.SessionToken'
}

aws_assume_role "${AWS_ACCOUNT_ID}" 'cicd'

Implement your script for GCP

#!/usr/bin/env bash
# @file cloud_authentication.sh

export GCP_WORKLOAD_IDENTITY_POOL_PROJECT_NUMBER='0000000000' # <= replace with your GCP project number where workload identity pool is setup
export GCP_PROJECT_ID='your-project-id' # <= replace with your GCP project ID where your service account is setup

gcp_impersonate() {
    export GOOGLE_APPLICATION_CREDENTIALS=".gcp_credentials"
    gcp_create_impersonation_credentials "$@"
    # verify the impersonation is working
    gcloud --quiet auth application-default print-access-token >/dev/null
}

gcp_create_impersonation_credentials() {
    local workload_identity_pool_project_number=$1
    local project_id=$2
    local service_account_name=$3
    local service_account="${service_account_name}@${project_id}.iam.gserviceaccount.com"

    local pool_provider
    pool_provider="$(gcp_pool_provider)"

    local credential_source_file=".gcp_credential_source"
    jwt_token >"${credential_source_file}"

    cat >"${GOOGLE_APPLICATION_CREDENTIALS}" <<EOT
{
    "type": "external_account",
    "audience": "//iam.googleapis.com/projects/${workload_identity_pool_project_number}/locations/global/workloadIdentityPools/cicd-pool/providers/${pool_provider}",
    "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
    "token_url": "https://sts.googleapis.com/v1/token",
    "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${service_account}:generateAccessToken",
    "credential_source": {
        "file": "${credential_source_file}",
        "format": {
            "type": "text"
        }
    }
}
EOT
}

gcp_pool_provider() {
    if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
        echo "gitlab"
    elif [[ -n "${CI+x}" ]]; then # GitHub
        echo "github"
    else
        exit 2
    fi
}

jwt_token() {
    if [[ -n "${GITLAB_CI+x}" ]]; then # GitLab
        echo "${CI_JOB_JWT_V2}"
    elif [[ -n "${CI+x}" ]]; then # GitHub
        echo "${ACTIONS_ID_TOKEN_REQUEST_TOKEN}"
    else
        exit 2
    fi
}

gcp_impersonate "${GCP_WORKLOAD_IDENTITY_POOL_PROJECT_NUMBER}" "${GCP_PROJECT_ID}" "cicd"

Conclusion

Now you should be able to deploy in AWS or GCP from GitLab or GitHub. I hope you will never ever need to save credentials into your CI/CD platform. If it is still the case, maybe you can store these into a Secret Manager or reproduce the mechanism above for your specific components.

Happy coding.

FAQ

It exists a method much simple for GitHub. Why not using it ?

It is because I like to know what is happening and how things are working.

Haha ! I found a security breach. Temporary credentials can be printed and used as-is. What can you do now ?

Yeah that is true. Just like the credentials saved in the CI/CD platform. But here they expire in 1 hour. So it is eventually more secured than the previous used method.

And Azure ?

Sorry, I’m not able to do it with this cloud provider. Because they are all similar, it should be adaptable.

Sources

If you need to go futher (and maybe simplify, enhance, fix all of theses) here are the links I used to write this article:

Commentaires :

A lire également sur le sujet :

Réussir sa transformation Cloud : facteurs ...

  • 13 janvier 2021
Soyons honnêtes : tous les projets Cloud ne réussissent pas du premier coup. De...

Sécuriser son déploiement Terraform sur AWS...

  • 29 avril 2021
Comme nous l’avons vu dans l'article précédent, il est difficile d’authentifier...

Devoteam Revolve obtient la compétence Sécu...

  • 01 septembre 2020
Depuis plusieurs années déjà, nous avons mobilisé nos équipes pour intégrer la ...

Définir le DevSecOps à travers les 12 princ...

  • 05 mai 2020
Dans la 3ème et dernière partie de cet article, je continue l’exploration des p...

A PROPOS DE L'AUTEUR

Alexandre Jannotta

Consultant et Manager, mais avant tout Développeur au sein de Devoteam Revolve, Alexandre est un touche à tout qui s'intéresse autant à la technique, qu'à l'humain. Agile et DevSecOps font partie de son mode de fonctionnement et il est toujours ouvert à la discussion. Papa d'une petite fille de 4 ans, lire et partager des connaissances sont pour lui un moyen de nourrir la curiosité et de créer la richesse.

Le blog reBirth

Nous luttons contre les raccourcis intellectuels, proposons des alternatives, challengeons les pratiques, partageons nos expériences et provoquons une réaction. En ce sens nous ENTREPRENONS et révélons les singularités.