How to remove all trace of a file from your git history (and automate it with BASH)

I am no web developer, but I have dabbled. I have built 2 websites, both offline now, for hobbies and/or potential businesses I wanted to try out.

Without going into their stories, as I was building these websites I became aware that it was best practice to put secret values (e.g. API keys) that one did not want to share to a Github repo into an .env file, and add that .env file to .gitignore. This keeps it from being added to the git history.

However, what if you are me, and didn’t add it to the .gitignore when you should have? Or what if you did finally add it, but you added it after you had already committed the .env file? The problem is that anything that was in the .env file when it was committed now lives in your git history, where some nefarious actor might sniff if out if your repo ever becomes public or shared with the bad actor in any way. .gitignore only ignores the file going forward, it doesn’t remove any of the references to it in the past git history.

What’s the solution? You can do an interactive rebase and remove all mentions of the .env file. Now, do this with care. Re-writing git history is no joke, and can cause lasting damage that is very difficult and even sometimes impossible to undo. You have to check everything very carefully, and be sure you are aware of the consequences of rebasing:

1. Things can break, and if you delete the wrong files they will be missing from your git history.
2. You may add files to your git history, if you are not careful.
3. This involves force-pushing to your Github repo, once you have over-written the history. If you have collaborators, you need to discuss this with them beforehand, because they will be very unhappy with you otherwise. You should also absolutely create a backup of your Github repo before you force-push, in case you want to undo this. You should also create a separate copy of the file you are removing as the very first thing you do, to avoid losing its contents.
4. You will overwrite the dates of the commits, and the new dates will be at the time of the rebase. The upside of this is that your git history will look very productive on the day of the rebase. However, messing up the history this way is generally undesired, but it is a cost of doing this.

To be clear, this technique works for any file in your git history, and is not limited to .env files and/or secret API keys. If you understand and are ok with the risks, let’s get to it. I am also going to share a simple BASH script to automate the rebase, so you don’t have to go through all of the (potentially very many) commits manually.

All of the instructions I give are based on a Linux system, and I’m not sure how it would translate to a Windows system. I will create a little dummy example to show how this work. Just substitute your file name and git repo for the example I walk through, and it should all work.

To start, we create an empty git repo, and add a couple of files.

my-folder$ git init
my-folder$ mkdir random
my-folder$ touch random/KEEP.md
my-folder$ git add random/KEEP.md
my-folder$ git commit -m "We keep this one"
my-folder$ touch random/REMOVE.md
my-folder$ git add random/REMOVE.md
my-folder$ git commit -m "What have I done"

Going forward, all of my commands will be run from my-folder , so I will remove it from the commands going forward. I create a couple more commits:

touch new.md
git add new.md
git commit -m "misc commit 1"
touch new2.md
git add new2.md
git commit -m "misc commit 2"

If we run git log, we get this:

Months have passed, and we realize we need to remove random/REMOVE.md from our git history.

The absolute first step, is to make a copy of the file you are removing, just in case. This can be a file that contains valuable information, and you want to have a copy somewhere before you delete all of its traces from your git repo.

The second step is to check the entire git history of your .env file.

git log -- random/REMOVE.md

This will log all of your git history that impacted that file:

If this is empty, it means this file does not exist in your git history. If so, congrats! You don’t need to do anything, and can just skip the rest of this tutorial. However, if there is a git history, you can see where the changes were made. This isn’t massively important, but we will return to this at the end as a check.

Next, we go looking for the ID of the commit we need to rebase to. We run the following:

git rev-list --all -- random/REMOVE.md

This returns a list of all of the commits that have impacted your file. In my case, it only returns the one commit. We want to take the last (i.e. bottom) commit in that list (which for me, is easy), and run the next line of code. It finds and logs the commit hash of the commit immediately previous to the provided commit, which is the commit hash we want to rebase to (i.e. we rebase to the commit just before we first added the file to the git history).

git log --pretty=format:"%H" -n 2 39ee5cd866490e3501df4e9a33f57233d0131e4b | tail -n 1

We can see it returned the commit hash from the first commit in my history, which is correct!

If this just returns the same commit hash as you put in, this means you added the file to the git history at the very first commit in the git repo. If so, in the line of code below, replace that commit hash with --root. This will rebase to before the initial commit. In our case:

git rebase -i 44b8045a0edf514fc3265d4e6f92569a3933da5

This will open an interactive rebase file, which in VS Code looks like this:

With the interactive rebase file open, you need to change all of the pick instances to edit or e . This can be done by opening up the file and manually editing them (in VS Code you can just click the name of the file at the top: XXX/.git/rebase-merge/git-rebase-todo), or by opening up another cmd window and running (replace XXX with the correct path):

sed -i 's/^pick/edit/' XXX/.git/rebase-merge/git-rebase-todo

Once the pick values have been set to edit (shorthand e ) in the interactive rebase file, you can save the file, and then ctrl+x to close the file from cmd line. This will automatically start the interactive rebase.

At this stage, we are at the first commit. Here, you can then delete the file you want to remove, git add . your changes, git commit them, and then git rebase --continue to go to the next step. However, if you have a large number of git commits to iterate through in your rebase, this can be quite tiresome, so here is the BASH script I wrote to automate this:

#!/bin/bash

# Number of times to run the command, 
# this should be equal to the number of commits you have to rebase
# NOTE: I highly recommend starting with one to make sure nothing breaks.
num_iterations=1

for ((i = 1; i <= num_iterations; i++)); do
    # Check if the file exists before removing it
    # NOTE: # Replace `random/REMOVE.md` with the path to your file from where you are currently located
    if [ -e random/REMOVE.md ]; then
        rm random/REMOVE.md
    fi
    
    # Be very careful here: I commit all via `.`, but you may want to be more selective.
    git add .
    # We unstage the bash script from the commit
    git reset delete.sh
    # We keep the same commit message as before the rebase for these changes
    # NOTE: I'm not sure if there's a way to keep the original date?
    git commit --amend --no-edit 
    # We continue to the next step in the rebase
    git rebase --continue
    
    # Wait for some time (make this larger if your computer is very slow / you are conservative).
    sleep 1
done

To run it, save it as delete.sh in your git repo (or some other name, but if you rename it please adjust the script to git reset your_bash_script.sh name). Then run chmod +x delete.sh to give it permissions, and then run ./delete.sh . This will run one iteration of the rebase. For the first iteration, it will throw you into the interactive debugger.

However, you can close this with ctlr+x immediately. After we close the interactive session, we get:

We can check that random/REMOVE.md has indeed been removed, at least for this commit, that new.md just got added, and that delete.sh is unmounted (green):

All has gone to plan, so I will increase the num_iterations variable in the script to 3, as I have 2 commits left to rebase and I want to show what happens if you specify “too many” iterations. If you specify too many, the redundant iterations will just throw warnings saying that there isn’t an on-going rebase ( fatal: No rebase in progress?), and you can easily kill the job at that point if it’s still running with ctrl+c. If I run it again with num_iterations=3, it automatically iterates through the remaining commits:

Now, we check that the file has been removed from the git history. Run the next two lines again to check:

git log -- random/REMOVE.md

git rev-list --all -- random/REMOVE.md

If they both are empty (as above), then congratulations! You have successfully rebased the local version of your git history, and removed that pesky file from your history. If not, check that the file was not introduced in your fist commit (if so, see above on using --root as the base commit of your rebase), and check that all of the paths in the bash script were correct.

Next, make sure you reintroduce your .env (or whatever random/REMOVE.md file) if you would like to do so from your copy, but make sure it’s included in your .gitignore.

Finally, now comes the truly dangerous part: force pushing to your Github (or other) repo.

Again, if you have collaborators, talk to them first. They will not thank you for causing an entire project’s worth of git conflicts. Secondly, make a backup copy of your unedited upstream/origin repo, in case you want to undo this.

Alright, moment of truth. In the code below replace your_branch_name (and maybe origin depending on where you’re pushing) as appropriate:

git push origin your_branch_name --force

And that is it. You have removed any trace of the targeted file from your git history. You have embraced the dark side of git with force pushes, I hope it doesn’t corrupt you.

I am sorry if this messed up your git history, people are screaming at you, and you feel a sense of budding Machiavellian power. But thus is the dark side, and I did warn you. But hopefully all is well, and now your secret files are safe and have been made invisible to your git history.

Hopefully you found this useful (and if you did, please click the applaud button thingy). Have a great day!