25 July 2023

Shift Left Security: The Git Backdoor in Deployment Pipelines

Cutting Corners

I’m convinced cutting corners is inextricably linked to how humans function. It’s like you’re attracted to cutting corners, instinctively following a desired path. Yes, I plead guilty. But before cutting a corner, I always go over the different solutions considering the tradeoffs of each option. That’s my process to find the right balance between cost and benefit.

Desired Path

From time to time, you stumble upon a solution you thought you’d well consider. When, a few weeks later, giving your solution a second thought, you’re fazed. That’s what happened to me in this case.

Get to the point

One of the first things you learn on AWS is to follow the standard security advice of applying least privilege. It’s a simple rule, but it can be challenging to implement. When creating an AWS CodePipeline a while ago, we came up with the following CloudFormation template describing the infrastructure:

    Type: AWS::CodePipeline::Pipeline
        - Region: eu-west-1
            Location: !Ref ArtefactBucketName
            Type: S3
      Name: some-project-pipeline
      RoleArn: !GetAtt CodePipelineServicenRole.Arn
        - Name: Source
            - Name: GitSource
                Category: Source
                Owner: ThirdParty
                Provider: GitHub
                Version: "1"
                Owner: !Ref GitHubOwner
                Repo: !Ref GitRepo
                Branch: !Ref GitBranch
                PollForSourceChanges: False
                OAuthToken: !Ref GitHubOAuthToken
                - Name: SourceZip
        - Name: Deploy
            - Name: DeployPersistantStack
                Category: Deploy
                Owner: AWS
                Provider: CloudFormation
                Version: "1"
                ActionMode: CREATE_UPDATE
                Capabilities: CAPABILITY_NAMED_IAM
                RoleArn: !GetAtt CloudFormationExecutionRole.Arn
                StackName: some-stack-cfn
                TemplatePath: BuildArtifactAsZip::cfn-template.yaml
                TemplateConfiguration: BuildArtifactAsZip::dist/config/cloudformation/stack-config.json
                - Name: BuildArtifactAsZip
              RunOrder: 1

    Type: AWS::IAM::Role
        Version: 2012-10-17
            - sts:AssumeRole
          Effect: Allow
              - cloudformation.amazonaws.com
      Path: /
        - arn:aws:iam::aws:policy/AdministratorAccess

The corner being cut in the code above is easy to find. A CloudFormationExecutionRole having AdministratorAccess doesn’t apply the least privilege principle. The reason for this shortcut is because CloudFormation stacks often do a lot of things. Applying the least privilege rule in this context means a long and well-thought-out policy. For simplicity and speed, I decided to cut a corner. The trade-off? Allowing the pipeline a tad more access than it should. No big deal, I could live with that.

A Dangerous Git Backdoor

To give a bit more context: security-wise, our AWS environment is a bit of a fortress: IDP/SSO, AWS WAF, AWS Security Hub, SAST/DAST, EDR and an endless list of tools to close the gates… We made AWS a safe place to go to.

Git repositories often live outside the AWS ecosystem. GitHub, GitLab and Bitbucket are most common to host any code. Are your Git repositories secured as well? If the answer is nah, then here’s some horror. If someone gains access to a Git repository, he could do the following if a pipeline is granted Administrator access:

  • Change a DNS Name to re-route traffic to a phishing site
  • Change your DeletionPolicy and throw away resources
  • Change an instance profile or a security group
  • Elevate IAM privileges and create an IAM user for himself
  • I think that’s already enough…

Here’s a visualization of the Git backdoor

Git Backdoor

Asking the same question again: can I live with Administrator access to my build pipeline? The answer is the opposite and it’s a big no-no. I’m feeling lucky I got this epiphany before things turned into a real issue.

How to shut the door?

The best way to close this security hole is to apply the least privilege rule to your pipeline. The downside of this approach is the introduction of extra work. Whenever someone changes a pipeline that incurs security context changes, they must update the pipeline’s policy first. Be prepared for people to be upset by this. Besides, it’s crucial to separate the concern for updating the pipeline policy and disconnect it from Git. Separating the responsibilities by disconnecting the pipeline from Git means it will require two different exploits to gain the same level of access, making changes for an exploit much smaller.

Git Least Privilege Access

If you are in the situation where your pipeline policy is too relaxed, you could consider starting with an intermediate solution. Start denying all IAM and Route 53 access and/or deny delete access for all resources. Although this is more secure, this approach has downsides. So, you only buy a bit more time to apply the least privilege principle.

Bonus Warning: Self-Mutating Pipelines

For clarity: with self-mutating pipelines I mean pipelines having a stage to update themselves. Although I was never a fan of pipelines maintaining themselves, in this case, be extra warned. Having such a pipeline listening to Git means that the Pipeline policy itself can be changed by a git push.

Self-Mutating Pipelines

Even if your self-mutating pipeline doesn’t have administrator access, this is easily changed for self-mutating pipelines using Git access. For that reason, remove self-updating logic from a pipeline. You can use makefiles that require client authentication in order to separate concerns of executing and updating pipelines.

Enjoy and until next time.

Subscribe to our newsletter

We'll keep you updated with more interesting articles from our team.

(about once a month)