I’m convinced cutting corners is inextricably linked to how human beings function. It’s like you’re attracted to cutting corners, like you instinctively follow the desired path. Yes, I plead guilty. I also cut corners. But before doing, I always go over the different solutions while in the meantime, I consider the tradeoffs of each option. That’s my process to find the right balance between cost and benefit.
From time to time however, you stumble upon a solution you thought you’d well consider. Then, a few weeks later, giving your solution a second thought, you’re fazed. That’s exactly what happened to me in this case.
Get to the point
One of the first things you learn on AWS is to follow the standard security advice of applying least privilege. It’s a simple rule, but it can be challenging to implement. When creating AWS CodePipelines a while ago, we came up with the following CloudFormation template describing the infrastructure:
DeliveryPipeline: Type: AWS::CodePipeline::Pipeline Properties: ArtefactStores: - Region: eu-west-1 ArtifactStore: Location: !Ref ArtefactBucketName Type: S3 Name: some-project-pipeline RoleArn: !GetAtt CodePipelineServicenRole.Arn Stages: - Name: Source Actions: - Name: GitSource ActionTypeId: Category: Source Owner: ThirdParty Provider: GitHub Version: "1" Configuration: Owner: !Ref GitHubOwner Repo: !Ref GitRepo Branch: !Ref GitBranch PollForSourceChanges: False OAuthToken: !Ref GitHubOAuthToken OutputArtifacts: - Name: SourceZip ... - Name: Deploy Actions: - Name: DeployPersistantStack ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CREATE_UPDATE Capabilities: CAPABILITY_NAMED_IAM RoleArn: !GetAtt CloudFormationExecutionRole.Arn StackName: some-stack-cfn TemplatePath: BuildArtifactAsZip::cfn-template.yaml TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json InputArtifacts: - Name: BuildArtifactAsZip RunOrder: 1 ... CloudFormationExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: Action: - sts:AssumeRole Effect: Allow Principal: Service: - cloudformation.amazonaws.com Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/AdministratorAccess
The corner being cut in the code above is easy to find. A
AdministratorAccess doesn’t apply the least privilege principle. The reason for this shortcut
is because CloudFormation stacks often do a lot of things. Applying the least privilege
rule in this context means a long and well-thought-out policy. For simplicity and speed, I decided
to cut a corner. The trade-off? Allowing the pipeline a tad more access than it should. No big
deal, I could live with that.
A Dangerous Git Backdoor
Security-wise, our AWS environment is a bit of a fortress: IDP/SSO, AWS WAF, AWS Security Hub, SAST/DAST, EDR and an endless list of tools to close the gates… We made AWS a safe place to go to.
Git repositories however, often live outside the AWS ecosystem. GitHub, GitLab and Bitbucket are most common to host your code. Are your Git repositories secured as well? If the answer is nah, then here’s some horror. If someone gains access to a Git repository, he could do the following if a pipeline is granted Administrator access to execute CloudFormation:
- Change a DNS Name to re-route traffic to a phishing site
- Change your DeletionPolicy and throw away resources
- Change an instance profile or a security group
- Elevate IAM privileges and create an IAM user for himself
- I think that’s already enough…
Here’s a visualization of the Git backdoor
Asking the same question again: can I live with Administrator access to execute CloudFormation in my build pipeline? The answer is the opposite and it’s a big no-no.
I’m feeling lucky I got this epiphany before things turned into a real issue.
How to shut the door?
The best way to close this security hole is to apply the least privilege rule to your Pipeline. The downside of this approach is the introduction of extra work. Whenever someone makes changes to the CloudFormation stack that incur security context changes, she/he will need to update the Pipeline’s policy first. Be prepared for people really upset by this.
If you are in the situation where your pipeline policy is too relaxed, you could consider starting with an intermediate solution. Start denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is more secure, this approach has downsides. So, you only buy a bit more time to apply the least privilege principle.
Bonus Warning: Self-maintaining Pipelines
For clarity: with self-maintaining pipelines I mean pipelines having a stage to update
themselves. Although I was never a fan of pipelines maintaining themselves, in this case, be extra
warned. Having such a pipeline listening to Git means that the Pipeline policy itself can be
changed by a
Even if your self-maintaining pipeline doesn’t have Administrator access to execute CloudFormation, this is easily changed for self-maintaining pipelines using Git access. For that reason, remove self-updating logic from a pipeline. You can use makefiles that require client authentication in order to separate concerns of executing and updating pipelines (see the image above).
Note: the same security holes can be created on AWS CodeBuild.
Enjoy and until next time.