The issue is that, even though you removed the files, they are still present in previous revisions. That’s the whole point of git, is that even if you delete something, you can still get it back by accessing the history.
What you are looking to do is called rewriting history, and it involved the
git filter-branch
command.
GitHub has a good explanation of the issue on their site. https://help.github.com/articles/remove-sensitive-data
To answer your question more directly, what you basically need to run is this:
git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch unwanted_folder' --prune-empty
This will remove all references to the files from the history of the repo.
Next, you’ll want to run this, to actually remove the files from the packfile.
git gc --aggressive --prune