Git Security Breach: How to Remove Secrets from Git History

It's every developer's nightmare: You're reviewing a pull request and notice that a Kubernetes Secret (or worse, a plaintext password) has been committed to your repository. Your heart races as you realize the implications—this sensitive information is now visible to everyone with repository access, and it's permanently stored in your Git history. But don't panic. With the right response strategy, you can contain the damage and prevent future incidents.

Understanding the Security Impact

Why This Is Critical

When secrets are committed to Git, they become:

Visible to all repository collaborators - Anyone with access can see them
Permanently stored in history - Even if deleted, they remain in previous commits
Accessible to CI/CD systems - Build pipelines can access the entire history
Potentially exposed to forks - Anyone who forked the repository has the secrets
Searchable in logs - Git history is searchable and auditable

Base64 encoding is NOT encryption. It's just encoding that can be easily decoded by anyone. Never rely on base64 encoding to protect sensitive information.

The Scope of the Problem

# Example of a compromised Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: database-secret
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=  # This is base64 encoded "password123"
  username: YWRtaW4=          # This is base64 encoded "admin"

What this means:

Anyone can decode these values using echo "cGFzc3dvcmQxMjM=" | base64 -d
The secrets are visible in the repository history
They're accessible to anyone with repository access

Emergency Response: The 3-Step Process

Step 1: Rotate the Compromised Secrets

Immediate Action Required: Assume all exposed secrets are compromised and rotate them immediately.

For Kubernetes Secrets

# Generate new secret values
NEW_PASSWORD=$(openssl rand -base64 32)
NEW_USERNAME="admin-$(date +%s)"

# Update the Kubernetes Secret
kubectl create secret generic database-secret \
  --from-literal=password="$NEW_PASSWORD" \
  --from-literal=username="$NEW_USERNAME" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart workloads using the secret
kubectl rollout restart deployment/database-app

For API Keys and Tokens

# Revoke old API keys
# - Go to your service provider's dashboard
# - Revoke the compromised key
# - Generate a new key
# - Update your applications with the new key

For Database Credentials

# Change database passwords
# - Connect to your database
# - Change the password for the compromised user
# - Update your applications with the new password

Always rotate secrets immediately, even if you're not sure they've been compromised. It's better to be safe than sorry.

Step 2: Remove Secrets from Git History

Critical: Simply deleting the file from the current commit is not enough. You must remove it from the entire Git history.

Option 1: Using `git filter-repo` (Recommended)

# Install git-filter-repo if not already installed
pip install git-filter-repo

# Remove the secret file from entire history
git filter-repo --path secret.yaml --invert-paths

# Force push to update remote repository
git push --force

Option 2: Using BFG Repo-Cleaner

# Download BFG Repo-Cleaner
wget https://repo1.maven.org/maven2/com/madgag/bfg/1.14.0/bfg-1.14.0.jar

# Remove the secret file
java -jar bfg-1.14.0.jar --delete-files secret.yaml

# Clean up and force push
git reflog expire --expire=now --all
git gc --prune=now --aggressive
git push --force

Option 3: Using `git filter-branch` (Legacy)

# Remove file from entire history
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch secret.yaml' \
  --prune-empty --tag-name-filter cat -- --all

# Force push to update remote
git push --force

Force pushing rewrites history and can cause issues for other developers. Coordinate with your team before force pushing.

Step 3: Notify and Coordinate with Team

Immediate Communication

# Notify team members immediately
# - Send urgent message to team chat
# - Explain what happened and what's being done
# - Provide instructions for team members

Instructions for Team Members

# Team members need to re-clone the repository
# 1. Backup any uncommitted work
# 2. Delete local repository
# 3. Clone fresh copy
# 4. Restore any uncommitted work

# Example commands:
cd ..
rm -rf your-repo
git clone https://github.com/your-org/your-repo.git
cd your-repo

Prevention: Building a Secure Workflow

1. Implement Secret Scanning

Using `git-secrets`

# Install git-secrets
git secrets --install

# Add patterns to scan for
git secrets --add 'password.*=.*'
git secrets --add 'api_key.*=.*'
git secrets --add 'secret.*=.*'

# Scan existing repository
git secrets --scan

Using Pre-commit Hooks

# Install pre-commit
pip install pre-commit

# Create .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

2. Use Proper Secret Management

For Kubernetes

# Use sealed secrets
kubectl create secret generic database-secret \
  --from-literal=password="$PASSWORD" \
  --dry-run=client -o yaml | kubeseal -o yaml > sealed-secret.yaml

# Use external secret operators
# - AWS Secrets Manager
# - Azure Key Vault
# - HashiCorp Vault

For Applications

# Use environment variables
export DATABASE_PASSWORD="secure-password"
export API_KEY="secure-api-key"

# Use configuration management tools
# - Ansible Vault
# - Terraform variables
# - Helm values

3. Implement Git Hooks

Pre-commit Hook

#!/bin/bash
# .git/hooks/pre-commit

# Check for common secret patterns
if git diff --cached --name-only | xargs grep -l "password.*="; then
    echo "ERROR: Potential password found in staged files"
    exit 1
fi

if git diff --cached --name-only | xargs grep -l "api_key.*="; then
    echo "ERROR: Potential API key found in staged files"
    exit 1
fi

exit 0

Pre-push Hook

#!/bin/bash
# .git/hooks/pre-push

# Scan for secrets before pushing
if git secrets --scan; then
    echo "ERROR: Secrets detected in repository"
    exit 1
fi

exit 0

Advanced Secret Management Strategies

1. Use Secret Templates

# Create secret template
cat > secret-template.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
  name: database-secret
type: Opaque
data:
  password: <BASE64_ENCODED_PASSWORD>
  username: <BASE64_ENCODED_USERNAME>
EOF

# Add to .gitignore
echo "secret.yaml" >> .gitignore
echo "secret-template.yaml" >> .gitignore

2. Implement Secret Rotation

#!/bin/bash
# rotate-secrets.sh

# Generate new secrets
NEW_PASSWORD=$(openssl rand -base64 32)
NEW_USERNAME="admin-$(date +%s)"

# Update Kubernetes Secret
kubectl create secret generic database-secret \
  --from-literal=password="$NEW_PASSWORD" \
  --from-literal=username="$NEW_USERNAME" \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart applications
kubectl rollout restart deployment/database-app

echo "Secrets rotated successfully"

3. Use External Secret Stores

AWS Secrets Manager

# Store secret in AWS Secrets Manager
aws secretsmanager create-secret \
  --name "database-credentials" \
  --secret-string '{"username":"admin","password":"secure-password"}'

# Retrieve in application
SECRET=$(aws secretsmanager get-secret-value \
  --secret-id "database-credentials" \
  --query SecretString --output text)

HashiCorp Vault

# Store secret in Vault
vault kv put secret/database \
  username=admin \
  password=secure-password

# Retrieve in application
vault kv get -field=password secret/database

Monitoring and Detection

1. Implement Continuous Monitoring

# Set up automated secret scanning
# - GitHub Secret Scanning
# - GitLab Secret Detection
# - Custom CI/CD checks

2. Use Security Tools

# Tools for secret detection
# - TruffleHog
# - GitLeaks
# - detect-secrets
# - GitGuardian

3. Regular Security Audits

# Regular security checks
# - Scan repository history
# - Review access permissions
# - Audit secret usage
# - Test incident response procedures

Rotate compromised secrets immediately - Assume they're compromised and generate new ones
Remove secrets from entire Git history - Use tools like git filter-repo or BFG
Notify and coordinate with your team - Ensure everyone is aware and takes appropriate action
Implement prevention measures - Use secret scanning, proper secret management, and security tools

Remember: The best defense against secret exposure is prevention. Implement proper secret management practices, use security tools, and educate your team about the risks and proper procedures.

Table of Contents