Scope

GitHub and GitHub Enterprise (GHE) servers may reject a push due to files exceeding a size limit. This document explains how to resolve the error by updating your local repository to remove the large files from your local branch.

Push Error When Files Exceed Size Limit

When pushing to a remote, you may encounter an error reporting that one or more files exceed the size limit.

remote: error: File foo is 501.00 MB; this exceeds GitHub Enterprise's file size limit of 100.00 MB

On https://github.com, pushing a file over 100 MiB triggers the above error. For GHE instances, 100 MiB is the default maximum file size, but the value may be configured by the administrator.

This document refers to files exceeding the limit as "large" files.

How to Resolve the Error

Overview of Steps

• Identify the unpushed commit or commits that added the large files.
• Remove the large files from your local branch.
• Push the updated branch.

In addition, it is a good idea to add a .gitignore entry to avoid unintentionally adding the large files to Git in the future.

Initial Inspection

Assess where your local branch is compared to the branch on the remote. In a terminal, fetch from the remote to ensure the remote-tracking branches are up to date.

$ git fetch origin

Run the following git log commands to see the status of your local branch compared to the remote-tracking branch.

# commits in current branch but not in origin/main
$ git log --oneline --graph origin/main..

# commits in origin/main but not in current branch
$ git log --oneline --graph ..origin/main

Note: this document assumes that you are working on the branch main and pushing that branch to a remote named origin. Adjust the names accordingly if that is not the case.

The first git log command shows commits that are not in the remote-tracking branch. The large files were introduced in one or more of these commits.

The output of the second git log command should be empty (i.e. your local branch contains all commits from the remote-tracking branch). If not, rebase onto the remote-tracking branch.

# Note: only do this if the second 'git log' command displays commits.
$ git rebase origin/main

# Confirm that this command no longer shows commits.
$ git log --oneline --graph ..origin/main

(If the git rebase command above fails due to a conflict and you are unable to resolve it, you can abort the rebase with git rebase --abort.)

Identify Where Large Files Were Added

The next step is to find out which commit or commits added the large files. If the last commit (i.e. the tip of the current branch, main) introduced the large files, you can use a simpler approach to remove the files from your branch.

If the first git log command in the last section displayed one line, then there is only one unpushed commit. The large files must have been added in that commit. You can continue to the "Remove files from last commit" section.

Otherwise, if the push error message identified only few large files that you think were likely introduced in the last commit, you can filter the git log output to that set of files and verify that the only output line is for the last commit.

$ git log --oneline --graph origin/main.. -- {large_file1} {large_file2}...
# The above should show only the last commit.  That is, it should match this:
$ git log --oneline --graph -n1 origin/main..

Alternatively, you can use a more general approach to identify all the files in the unpushed commits that exceed a given threshold.

$ git rev-list --reverse --in-commit-order --objects \
    ^origin/main HEAD | \
  git cat-file \
    --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  grep -v '^tree ' | \
  awk '$1 != "blob" || $3 > NNNNNNNNN'

In the above command, NNNNNNNNN is a placeholder for the file size limit in bytes. 104857600, for example, would be the value for the GHE default limit of 100 MiB.

Example

Consider the following local branch, which is three commits ahead of origin/main.

$ git log --oneline --graph origin/main..

* 311e8f6 (HEAD -> main) add largefile2
* 4fc0eca add largefile1
* 45cea3d add script

Running this pipeline identifies the blobs over 100 MiB.

$ git rev-list --reverse --in-commit-order --objects \
    ^origin/main HEAD | \
  git cat-file \
    --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  grep -v '^tree ' | \
  awk '$1 != "blob" || $3 > 104857600'

commit 45cea3dda3ce567db8989fba7ff643abe318c2d3 221
commit 4fc0ecaae9bd7479256b0b687f2601227e16bbb2 225
blob 200859f0fa4b56a5ed14790ddf094ddb82b3a648 209715200 largefile1
commit 311e8f699d9e4d7daab35caaad45608bdff1ba55 225
blob 2f0c2fa6d740a5ed208b2245f6ad18ab3616c9cc 209715100 largefile2

Each blob line identifies a file over 100 MiB. The commit line before it is the commit that added the file. In the above case, the tip commit of main (311e8f6) and its parent (4fc0eca) added files over 100 MiB. The initial commit on top of origin/main (45cea3d) did not add a large file.

Removing Large Files

Now that you have determined if the large files were added only in the last commit, you are ready to decide on the strategy to remove them.

Remove Files From Last Commit

If the large files were added in the last commit, you can amend the commit.

First, remove the files from Git's index, leaving the files untracked in the working tree.

$ git rm --cached -- {large_file1} {large_file2}...

Consider adding a .gitignore pattern. See the "Ignoring large files" section below.

Then create a new commit, replacing the last commit.

$ git commit --amend --no-edit

(If you prefer, you can create this commit with the RStudio Git interface rather than running the above git commit command. Just make sure to select "Amend previous commit" when the commit window pops up.)

In most cases, you can now continue to the "Push the updated branch" section. If the above git commit command failed with "You asked to amend the most recent commit, but doing so would make it empty", it means that the last commit did not make any changes aside from adding the large files. In that case, you can instead just drop the last commit.

$ git reset HEAD~

Remove Files From Other Commits

If the large files were added by commits other than the last one, there are a few different approaches to removing it. A common option would be an interactive git rebase, which allows precise editing of the history. This section presents a less fine-grained strategy that uses git reset.

Move the tip of the current branch to origin/main.

$ git reset origin/main

All of the changes, including the large files, that were in the previous tip are now untracked files.

Now you can selectively call git add on the untracked files, making sure to not include the large files. Consider adding a .gitignore pattern (see the "Ignoring large files" section below).

$ git add {non_large_file1} {non_large_file2}...
$ git commit

(If you prefer, you can use the RStudio Git interface to stage the changes and create the commit. The important part is to not stage the large files.)

Push the Updated Branch

At this point, the large files have been removed from your local branch. You can confirm this by running the git rev-list ... pipeline from the "Identify where large files were added" section. The output should no longer show any blob lines.

Push the updated branch to the remote.

$ git push origin HEAD

Ignoring Large Files

After removing large files from your branch, you will want to make sure that they do not get added again. You can tell Git to ignore them (i.e. no longer list them as untracked files or consider them candidates for git add) via .gitignore entry.

A few examples are below. These assume you are editing the .gitignore at the top-level of the repository. See Ignoring Files with .gitignore and the gitignore manpage for more details.

• ignore all files that end with .iph

  *.iph

• ignore any files under an output directory at any depth in the repository

  output/

• ignore any files under the output directory at the top-level of the repository

  /output/

Once you have modified .gitignore, commit those changes.

$ git add .gitignore
$ git commit -m 'update .gitignore'

Checking for large untracked files

In some cases, you may have many untracked files and want to know which ones will run into issues with the maximum file size limit if you start to track them. You can see a list of your largest untracked files by running the following command.

$ git ls-files -oz --exclude-standard | du --files0-from=- -b | \
  sort -gr | head -n20 | numfmt --to=iec-i