A significantly faster version:
--------------------------------------------------------------------------
-git filter-branch --index-filter 'git update-index --remove filename' HEAD
+git filter-branch --index-filter 'git rm --cached filename' HEAD
--------------------------------------------------------------------------
Now, you will get the rewritten history saved in HEAD.