22 November 2022

Shift Left Security: The Git Backdoor in AWS Codepipeline and Codebuild

Cutting Corners

I’m convinced cutting corners is inextricably linked to how human beings function. It’s like you’re attracted to cutting corners, like you instinctively follow the desired path. Yes, I plead guilty. I also cut corners. But before doing, I always go over the different solutions while in the meantime, I consider the tradeoffs of each option. That’s my process to find the right balance between cost and benefit.

Desired Path

From time to time however, you stumble upon a solution you thought you’d well consider. Then, a few weeks later, giving your solution a second thought, you’re fazed. That’s exactly what happened to me in this case.

Get to the point

One of the first things you learn on AWS is to follow the standard security advice of applying least privilege. It’s a simple rule, but it can be challenging to implement. When creating AWS CodePipelines a while ago, we came up with the following CloudFormation template describing the infrastructure:

  DeliveryPipeline:
    Type: AWS::CodePipeline::Pipeline
    Properties:
      ArtefactStores:
        - Region: eu-west-1
          ArtifactStore:
            Location: !Ref ArtefactBucketName
            Type: S3
      Name: some-project-pipeline
      RoleArn: !GetAtt CodePipelineServicenRole.Arn
      Stages:
        - Name: Source
          Actions:
            - Name: GitSource
              ActionTypeId:
                Category: Source
                Owner: ThirdParty
                Provider: GitHub
                Version: "1"
              Configuration:
                Owner: !Ref GitHubOwner
                Repo: !Ref GitRepo
                Branch: !Ref GitBranch
                PollForSourceChanges: False
                OAuthToken: !Ref GitHubOAuthToken
              OutputArtifacts:
                - Name: SourceZip
        ...
        - Name: Deploy
          Actions:
            - Name: DeployPersistantStack
              ActionTypeId:
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: "1"
              Configuration:
                ActionMode: CREATE_UPDATE
                Capabilities: CAPABILITY_NAMED_IAM
                RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                StackName: some-stack-cfn
                TemplatePath: BuildArtifactAsZip::cfn-template.yaml
                TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json
              InputArtifacts:
                - Name: BuildArtifactAsZip
              RunOrder: 1
  ...

  CloudFormationExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          Action:
            - sts:AssumeRole
          Effect: Allow
          Principal:
            Service:
              - cloudformation.amazonaws.com
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AdministratorAccess

The corner being cut in the code above is easy to find. A CloudFormationExecutionRole having AdministratorAccess doesn’t apply the least privilege principle. The reason for this shortcut is because CloudFormation stacks often do a lot of things. Applying the least privilege rule in this context means a long and well-thought-out policy. For simplicity and speed, I decided to cut a corner. The trade-off? Allowing the pipeline a tad more access than it should. No big deal, I could live with that.

A Dangerous Git Backdoor

Security-wise, our AWS environment is a bit of a fortress: IDP/SSO, AWS WAF, AWS Security Hub, SAST/DAST, EDR and an endless list of tools to close the gates… We made AWS a safe place to go to.

Git repositories however, often live outside the AWS ecosystem. GitHub, GitLab and Bitbucket are most common to host your code. Are your Git repositories secured as well? If the answer is nah, then here’s some horror. If someone gains access to a Git repository, he could do the following if a pipeline is granted Administrator access to execute CloudFormation:

  • Change a DNS Name to re-route traffic to a phishing site
  • Change your DeletionPolicy and throw away resources
  • Change an instance profile or a security group
  • Elevate IAM privileges and create an IAM user for himself
  • I think that’s already enough…

Here’s a visualization of the Git backdoor

Git Backdoor

Asking the same question again: can I live with Administrator access to execute CloudFormation in my build pipeline? The answer is the opposite and it’s a big no-no.

I’m feeling lucky I got this epiphany before things turned into a real issue.

How to shut the door?

The best way to close this security hole is to apply the least privilege rule to your Pipeline. The downside of this approach is the introduction of extra work. Whenever someone makes changes to the CloudFormation stack that incur security context changes, she/he will need to update the Pipeline’s policy first. Be prepared for people really upset by this.

If you are in the situation where your pipeline policy is too relaxed, you could consider starting with an intermediate solution. Start denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is more secure, this approach has downsides. So, you only buy a bit more time to apply the least privilege principle.

Bonus Warning: Self-maintaining Pipelines

For clarity: with self-maintaining pipelines I mean pipelines having a stage to update themselves. Although I was never a fan of pipelines maintaining themselves, in this case, be extra warned. Having such a pipeline listening to Git means that the Pipeline policy itself can be changed by a git push.

Self-maintinging

Even if your self-maintaining pipeline doesn’t have Administrator access to execute CloudFormation, this is easily changed for self-maintaining pipelines using Git access. For that reason, remove self-updating logic from a pipeline. You can use makefiles that require client authentication in order to separate concerns of executing and updating pipelines (see the image above).

Note: the same security holes can be created on AWS CodeBuild.

Enjoy and until next time.

Subscribe to our newsletter

We'll keep you updated with more interesting articles from our team.

(about once a month)