Bootstrap your Google Cloud Foundation with Terraform and Gitlab CI

Introduction

The Cloud Foundation Toolkit is a set of templates written for Google Deployment Manager or HashiCorp Terraform, with Google best practices built into it to provide enterprise-grade Infrastructure as Code for your deployments on Google Cloud Platform.

This toolkit includes over 45 different Terraform modules covering many parts to build an end-to-end solution. Not only modules for the basic components like Google Storage buckets, Pub/Sub topics or VMs are available, but modules for more complex units, e.g. a project factory or network topologies, are available at Github.

Along with the foundation modules, there are also two end-to-end examples. One example shows how the CFT Terraform modules can be composed to build a secure GCP foundation, following the Google Cloud security foundations guide, and is intended to form a starting for building your own foundation.

The example foundation consists of a few steps where the first one is executed manually. From then onwards, the Terraform code is deployed automatically by leveraging either Cloud Build or Jenkins.

In this post, I would like to focus on the initial bootstrap step of the foundation. The other steps build a complete structure of folder, projects and resource and are maybe opinionated. It gives you an idea and best practices on structuring your organisation, but perhaps you would like to organize things differently. The first manual bootstrap step, on the other hand, is actually a very interesting one and is a necessity no matter how you want to organise your Google Cloud infrastructure.

Instead of using those proposed automation tools, we will explore how Gitlab CI can be used as our preferred CI/CD tool.

Bootstrapping the Cloud Foundation

This stage executes the CFT Bootstrap module, which bootstraps an existing GCP organization, creating all the required GCP resources & permissions to start using the Cloud Foundation Toolkit (CFT). For CI/CD pipelines, we will use Gitlab CI instead of Cloud Build or Jenkins, used in the example foundation.

First, I cloned the example foundation, removed everything related to Cloud Build and Jenkins, and kept the initial bootstrap code. Next, I’ve added a new module to create the Gitlab CI runners. The result is available on Github.

The bootstrap step takes the following variables:

org_id - your GCP organization id
billing_account - your GCP billing account
gitlab_group_path - your Gitlab group path

and includes:

The cft-seed project, which contains:
- a Terraform state bucket
- a service account used by Terraform to create new resources in GCP
The cft-cicd project, which contains:
- a GCE Instance configured as a Gitlab Runner
- a service account for the Gitlab Runner

Seed and CICD projects

A so-called seed project cft-seed is created. This project holds a GCS bucket to store the Terraform state of the infrastructure and the encryption configuration to keep the state protected. It also houses a highly privileged Service Account with the necessary permissions to create or modify infrastructure, enabling IaC deployments.

Our preferred CI/CD tool to run our infrastructure deployments is deployed in a separate, tightly controlled project cft-cicd. It hosts a Service Account that’s used to run the Compute Engine instances for the Gitlab Runners. When the Gitlab Runner executes Terraform jobs on those machines, Terraform will use that service account to authenticate to Google. This means you don’t have to configure secrets or credentials in Gitlab CI, making it more secure. The CI/CD service account is given permissions to impersonate the Terraform service account in the seed project.

Using two different projects and two different service accounts gives a clear separation of concerns. While Terraform is used as the IaC tool and has its own requirements, the deployment of that IaC is the responsibility of the CI/CD pipeline.

Deployment pipeline architecture

The diagram below shows the foundation deployment pipeline for our organisation. Terraform code for the infrastructure resources is stored in Git repositories on Gitlab. Code changes will trigger the Gitlab CI pipeline that starts deploying the Terraform code executed by the Gitlab Runner on Google Cloud.

In the example, the Gitlab Runner is configured to execute the pipeline steps with Docker containers on a single host. When the organisation grows, and more concurrent builds are required, autoscaling of the workers can easily be achieved by configuring the runner with Docker Machine.

Using this architecture in Google Cloud allows a clean authentication and authorization process for Google Cloud APIs. The Gitlab Runners use a custom service account that doesn’t have permission to create new infrastructure and that instead impersonates the Terraform service account to deploy infrastructure.

Gitlab CI pipelines

In Gitlab CI/CD, runners run the code defined in .gitlab-ci.yml. A runner is a lightweight, highly scalable agent that picks up a CI job through the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. There are three types of runners:

Shared runners are available to all groups and projects in a GitLab instance.
Group runners are available to all projects and subgroups in a group.
Specific runners are associated with specific projects. Typically, specific runners are used for one project at a time.

In our case, the Gitlab Runner for our Terraform deployment pipeline is registered at a dedicated Gitlab Group hosting all the different Git repositories for the Terraform Code. By doing so, the runner is available for all projects and subgroups in that group.

Besides the Google Cloud projects and the Gitlab Runner, the bootstrap step will also create some CI/CD variables for your Gitlab Group. Defining them at the group level makes them available for all projects, just like the runner.

Example pipeline configuration

The following yaml snippet is a lightweight .gitlab-ci.yml example for running the Terraform Code:

image:
  name: hashicorp/terraform:0.15.3
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

.tf-init: &tf-init
  - terraform --version
  - terraform init -reconfigure

stages:
- validate
- plan
- apply

Validate:
  stage: validate
  tags: ["gcp", "terraform"]
  script:
    - *tf-init
    - terraform validate
    - terraform fmt -check=true
  only:
    - branches

Plan:
  stage: plan
  tags: ["gcp", "terraform"]
  script:
    - *tf-init
    - terraform plan
  only:
    - main

Apply:
  stage: apply
  tags: ["gcp", "terraform"]
  script:
    - *tf-init
    - terraform apply -auto-approve
  when: manual
  only:
    - main

When using this configuration, the Terraform manifests are validated for each branch created and will only be planned and applied after approval when pushed to the main branch.

Service Account Impersonation

As the runner uses its own service account, we need to find a way to make sure it will impersonate the Terraform service account when creating or modifying our cloud infrastructure resources.
The Terraform Google provider has the option to impersonate such a service account. By setting the impersonate_service_account attribute in the provider configuration, all Google API calls are executed on behalf of our Terraform service account.

The following HCL snippet has to be available in each repository.

locals {
  tf_sa = var.terraform_service_account
}

provider "google" {
  impersonate_service_account = local.tf_sa
}

provider "google-beta" {
  impersonate_service_account = local.tf_sa
}

Going further

The Cloud Foundation Toolkit provides a series of reference templates for Terraform and can be used off-the-shelf to build a repeatable enterprise-ready foundation in Google Cloud quickly. It includes 45 Terraform modules, and besides those also two end-to-end examples are available. One of them is an Example Foundation holding several steps, starting with bootstrapping a seed and a CICD project.

While the Example Foundation leverages Cloud Build or Jenkins as the CI/CD tool, in this post, we took a different automation tool, Gitlab CI, while keeping all the recommended Google Cloud best practices.

Now that the base of our foundation is bootstrapped, we could leverage Gitlab CI to build the rest of our Google Cloud organisation. Whether you continue with the other steps in the example or build a completely different setup, everything is in place to apply Terraform code automatically. The Gitlab Runner and required CI variables are configured at the Gitlab Group, meaning every repository within the group can use the runner.

See also:

Provision a Multi-Region k3s cluster on Google Cloud with Terraform

References: