Terraform configuration for the Federalist platform.
- Terraform 0.13.2
- Access to AWS, cloud.gov, and CircleCI credentials for each environment
Future goals:
- replace local Terraform dependency with Docker
- add pre-commit hooks to validate, lint, scan, etc
- add Makefile to standardize commands
- add auto-generated module documentation
- add automation around bootstrapping
To contain the blast radius of any changes, all credentials used or created by Terraform should only have the fewest permissions necessary within the target environments.
Groups of related configuration are organized into root and shared Terraform modules, which are pretty much exactly like they sound. Root modules correspond to environments or other groups of configuration and are isolated from each other with distinct Terraform state files while shared modules provide a way to re-use common configuration. When Terraform is run, it is always for a single root module.
backend
: Configures the Terraform backend (Terraform S3 backend). Only run once, locally with MFA-protected admin credentials, when bootstrapping the entire repository. See (Bootstrapping the backend).bootstrap-env
: Bootstraps a new environment/AWS account. Only run once, locally with MFA-protected admin credentials, when adding a new environment/AWS account. See (Bootstrapping environments).staging
: Configures the staging environmentproduction
: Configures the production environment
ecr
: AWS ECR instance forfederalist-garden-build
docker imagesqueue
: AWS SQS queue and associated AWS CloudWatch alarmsshared
: Common infrastructure configuration for all environmentssns
*: AWS SNS topic used for AWS CloudWatch alarms.
*Note: When first creating SNS resources, the subscription to send emails to [email protected]
must be created manually in the console as it is currently not supported by Terraform.
While we want to simplify our configuration as much as possible and minimize the number of credentials, we have to work with the following constraints:
- Each Federalist environment may have resources in both AWS Commercial and AWS GovCloud
- Each Federalist environment requires isolated AWS accounts
- AWS cross-account permissions do not work between AWS Commercial and AWS GovCloud
- Best practice is to maintain "admin" accounts seperate from the accounts which contain the resources managed by Terraform
- Single Terraform backend
The result is that we have 3 AWS account-pairs:
- Admin
- Production
- Staging
each with a Commercial and GovCloud account and we only need 3 sets of credentials:
- Backend
- Commercial
- GovCloud
regardless of the number of environments.
In the event that we wish to add another non-production environment, we will re-use the Staging environment, making sure to tag resources appropriately.
To achieve this we:
-
Use a single Terraform backend that lives in the AWS Admin GovCloud account and is configured in
./terraform/backend
root module (see Bootstrapping the backend). Credentials for the IAM user with access to this backend is required for every root module. -
configure a
terraform-user-role
in each environment account, with permissions to manage Terraform resources within the account and in each Admin account we configure aterraform-user
with permissions to assume the specific role in each environment account for the same platform (Commercial/GovCloud) (see Bootstrapping environments).
- clone the repository:
git clone [email protected]:18F/federalist-infra.git
- enter the repository directory:
cd federalist-infra
- create the shared backend configuration by creating a copy of
./terraform/.backend-config.tfvars.example
named./terraform/.backend-config.tfvars
and populating the credentials for theterraform-backend
user in AWS Admin GovCloud - for each desired environment (
staging
orproduction
):- enter the environment directory:
cd terraform/<environment>
- create environment-specific credentials by creating a copy of the desired environment's
.secrets.auto.tfvars.example
file named.secrets.auto.tfvars
and populate with appropriate values - initialize terraform:
terraform init -backend-config=../.backend-config.tfvars
- enter the environment directory:
Within each environment, you should be able run terraform plan
to see the potential effects of any changes.
Modifying Terraform configuration can be tricky business because we can't fully test the changes before actually applying them and we want any changes to happen as part of the CI/CD pipeline. To mitigate the risks as much as we can, we will do the following for every change:
- Validate with
terraform validate
(enforced by CI) - Format with
terraform format
(enforced by CI) - Run
terraform plan
locally to inspect the potential changes (also run in CI) - Thoroughly review pull requests
- (TODO) Lint with TFLint
- (TODO) Security Scan with TFSEC
Changes in the Terraform configuration are applied using Github Actions according to the terraform
workflow. When a Github pull request is created against the default branch (main
), the terraform
job is run for each configured environment (staging
, production
) with the results of the corresponding terraform plan
added as a comment to the pull request. This output should be reviewed in detail before the pull request is approved. terraform apply
is only run when the pull request is merged to the default branch.
Note: Only the staging
and production
root modules should be run in CI.
These credentials correspond to the terraform-backend
user configured in the AWS Admin GovCloud account:
BACKEND_AWS_ACCESS_KEY_ID
BACKEND_AWS_SECRET_ACCESS_KEY
BACKEND_AWS_DEFAULT_REGION
These credentials correspond to the terraform-user
user configured in the specified platform AWS Admin account:
TF_VAR_aws_access_key_govcloud
TF_VAR_aws_secret_key_govcloud
TF_VAR_aws_access_key_commercial
TF_VAR_aws_secret_key_commercial
These correspond to the space deployer credentials in each environment (federalist-staging-deployer-circle
, federalist-production-deployer-circle
):
CF_USER_STAGING
CF_PASSWORD_STAGING
CF_USER_PRODUCTION
CF_PASSWORD_PRODUCTION
CircleCI API Key
TF_VAR_circleci_api_key
To ensure that we can use least-privilege credentials when provisioning resources with Terraform, it is necessary to have some bootstrapping steps that are run once with privileged credentials in a local environment.
This only needs to be done once, and only when starting completely from scratch. Since the Terraform configuration includes the resources user for storing the Terraform state, we must create the resources first using local state, then add the backend configuration which makes use of those resources. Terraform is smart enough to recognize when the backend configuration changes.
Requirements:
- Admin credentials for the AWS Admin GovCloud account
Working locally on your GSA machine, perform the following steps:
cd terraform/backend
- ensure admin credentials are in environment (eg
aws-vault exec <your admin profile> bash
) - comment out the
backend
block in./main.tf
terraform init
terraform plan
- verify the plan is correct
terraform apply
- make note of the backend access key and secret in the outputs
- create the backend config file as described in Getting started with the credentials from the previous step
- uncomment the
backend
block in./main.tf
terraform init -backend-config=../.backend-config.tfvars
terraform plan
- verify no changes need to be made
- ensure
.backend-config.tfvars
will NOT be checked into version control
Each AWS Admin account must contain a dummy user with the ability to assume a role in a target account that will allow it manage Terraform resources. This allows the use of only one set of credentials for each platform (commercial, govcloud) and limits the required permissions.
Requirements:
- Console access to AWS Admin Commercial and GovCloud accounts
In the AWS Console for each platform
- create a user named
terraform-user
- add the inline policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resources": [<staging_role_arn>, <production_role_arn>, ...]
}
]
}
where each <role_arn>
corresponds to the assume role arn created when bootstrapping an environment. These can be added now if they are known, or they can be added as created later on. Ex. For GovCloud, they will look like: arn:aws-us-gov:iam::<account id>:role/terraform-user-role
.
Before using Terraform to manage resources for a Federalist environment, we need to create a role with appropriate permissions in each AWS account associated with the environment. Since a Federalist environment may need to manage resources in both Commercial and GovCloud accounts AND accounts on different AWS platforms cannot interact, we must run this step for each one. In the steps below, <platform>
can be either commercial
or govcloud
and <env>
can be staging
or production
. This only needs to be done once when creating an environment completely from scratch OR when updating the Terraform role permissions.
Requirements:
- Admin credentials for the target AWS account
- Console access to AWS Admin account for the target platform
- The backend credentials file as described in Getting started.
Working locally on your GSA machine, perform the following steps:
cd terraform/bootstrap-env
- remove existing backend configuration with
rm -rf .terraform
- ensure admin creds are in environment (eg
aws-vault exec <your admin profile> bash
) terraform init -backend-config=../.backend-config.tfvars -backend-config="key=bootstrap-<env>-<platform>/terraform.tfstate"
terraform plan -var="aws_platform=<platform>"
- verify the plan is correct
terraform apply -var="aws_platform=<platform>"
- make note of the outputted value for
assume_role_arn
(you will need this for 2 subsequent steps) - In the AWS Console for the AWS Admin account responsible for the target platform, add the outputted role arn to allowed resources in the trust policy for the
terraform-user
. - Create a new root module for the environment by copy/pasting an existing one, making sure to update all variables as appropriate including
aws_assume_role_arn_<platform>
outputted above
Before commiting your changes, please be sure the configuration and format is valid by running terraform validate
and terraform format
. In the future, pre-commit hooks will be added to ensure this happens automatically.
This project is in the worldwide public domain. As stated in CONTRIBUTING:
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.