NAME
----
-git-fast-import - Backend for fast Git data importers.
+git-fast-import - Backend for fast Git data importers
SYNOPSIS
with the newly imported data.
The fast-import backend itself can import into an empty repository (one that
-has already been initialized by gitlink:git-init[1]) or incrementally
+has already been initialized by linkgit:git-init[1]) or incrementally
update an existing populated repository. Whether or not incremental
imports are supported from a particular foreign source depends on
the frontend program in use.
Dumps the internal marks table to <file> when complete.
Marks are written one per line as `:markid SHA-1`.
Frontends can use this file to validate imports after they
- have been completed.
+ have been completed, or to save the marks table across
+ incremental runs. As <file> is only opened and truncated
+ at checkpoint (or completion) the same path can also be
+ safely given to \--import-marks.
+
+--import-marks=<file>::
+ Before processing any input, load the marks specified in
+ <file>. The input file must exist, must be readable, and
+ must use the same format as produced by \--export-marks.
+ Multiple options may be supplied to import more than one
+ set of marks. If a mark is defined to different values,
+ the last file wins.
--export-pack-edges=<file>::
After creating a packfile, print a line of data to
This information may be useful after importing projects
whose total object set exceeds the 4 GiB packfile limit,
as these commits can be used as edge points during calls
- to gitlink:git-pack-objects[1].
+ to linkgit:git-pack-objects[1].
--quiet::
Disable all non-fatal output, making fast-import silent when it
spaces in their name, or early termination of fast-import when it encounters
unexpected input.
+Stream Comments
+~~~~~~~~~~~~~~~
+To aid in debugging frontends fast-import ignores any line that
+begins with `#` (ASCII pound/hash) up to and including the line
+ending `LF`. A comment line may contain any sequence of bytes
+that does not contain an LF and therefore may be used to include
+any detailed debugging information that might be specific to the
+frontend and useful when inspecting a fast-import data stream.
+
Date Formats
~~~~~~~~~~~~
The following date formats are supported. A frontend should select
+
An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git
parser is accurate, but a little on the lenient side. It is the
-same parser used by gitlink:git-am[1] when applying patches
+same parser used by linkgit:git-am[1] when applying patches
received from email.
+
Some malformed strings may be accepted as valid dates. In some of
+
Frontends should prefer the `raw` format if the source material
already uses UNIX-epoch format, can be coaxed to give dates in that
-format, or its format is easiliy convertible to it, as there is no
+format, or its format is easily convertible to it, as there is no
ambiguity in parsing.
`now`::
This particular format is supplied as its short to implement and
may be useful to a process that wants to create a new commit
right now, without needing to use a working directory or
-gitlink:git-update-index[1].
+linkgit:git-update-index[1].
+
If separate `author` and `committer` commands are used in a `commit`
the timestamps may not match, as the system clock will be polled
This command is optional and is not needed to perform
an import.
+`progress`::
+ Causes fast-import to echo the entire line to its own
+ standard output. This command is optional and is not needed
+ to perform an import.
+
`commit`
~~~~~~~~
Create or update a branch with a new commit, recording one logical
data
('from' SP <committish> LF)?
('merge' SP <committish> LF)?
- (filemodify | filedelete | filedeleteall)*
- LF
+ (filemodify | filedelete | filecopy | filerename | filedeleteall)*
+ LF?
....
where `<ref>` is the name of the branch to make the commit on.
and are not interpreted by Git. Currently they must be encoded in
UTF-8, as fast-import does not permit other encodings to be specified.
-Zero or more `filemodify`, `filedelete` and `filedeleteall` commands
+Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`
+and `filedeleteall` commands
may be included to update the contents of the branch prior to
creating the commit. These commands may be supplied in any order.
-However it is recommended that a `filedeleteall` command preceed
-all `filemodify` commands in the same commit, as `filedeleteall`
+However it is recommended that a `filedeleteall` command precede
+all `filemodify`, `filecopy` and `filerename` commands in the same
+commit, as `filedeleteall`
wipes the branch clean (see below).
+The `LF` after the command is optional (it used to be required).
+
`author`
^^^^^^^^
An `author` command may optionally appear, if the author information
`from`
^^^^^^
-Only valid for the first commit made on this branch by this
-fast-import process. The `from` command is used to specify the commit
-to initialize this branch from. This revision will be the first
-ancestor of the new commit.
-
-Omitting the `from` command in the first commit of a new branch will
-cause fast-import to create that commit with no ancestor. This tends to be
-desired only for the initial commit of a project. Omitting the
-`from` command on existing branches is required, as the current
-commit on that branch is automatically assumed to be the first
-ancestor of the new commit.
+The `from` command is used to specify the commit to initialize
+this branch from. This revision will be the first ancestor of the
+new commit.
+
+Omitting the `from` command in the first commit of a new branch
+will cause fast-import to create that commit with no ancestor. This
+tends to be desired only for the initial commit of a project.
+If the frontend creates all files from scratch when making a new
+branch, a `merge` command may be used instead of `from` to start
+the commit with an empty tree.
+Omitting the `from` command on existing branches is usually desired,
+as the current commit on that branch is automatically assumed to
+be the first ancestor of the new commit.
As `LF` is not valid in a Git refname or SHA-1 expression, no
quoting or escaping syntax is supported within `<committish>`.
+
The reason fast-import uses `:` to denote a mark reference is this character
is not legal in a Git branch name. The leading `:` makes it easy
-to distingush between the mark 42 (`:42`) and the branch 42 (`42`
+to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
or `refs/heads/42`), or an abbreviated SHA-1 which happened to
consist only of base-10 digits.
+
* A complete 40 byte or abbreviated commit SHA-1 in hex.
* Any valid Git SHA-1 expression that resolves to a commit. See
- ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
+ ``SPECIFYING REVISIONS'' in linkgit:git-rev-parse[1] for details.
The special case of restarting an incremental import from the
current branch value should be written as:
`merge`
^^^^^^^
-Includes one additional ancestor commit, and makes the current
-commit a merge commit. An unlimited number of `merge` commands per
+Includes one additional ancestor commit. If the `from` command is
+omitted when creating a new branch, the first `merge` commit will be
+the first ancestor of the current commit, and the branch will start
+out with no files. An unlimited number of `merge` commands per
commit are permitted by fast-import, thereby establishing an n-way merge.
However Git's other tools never create commits with more than 15
additional ancestors (forming a 16-way merge). For this reason
it is suggested that frontends do not use more than 15 `merge`
-commands per commit.
+commands per commit; 16, if starting a new, empty branch.
Here `<committish>` is any of the commit specification expressions
also accepted by `from` (see above).
In both formats `<path>` is the complete path of the file to be added
(if not already existing) or modified (if already existing).
-A `<path>` string must use UNIX-style directory seperators (forward
+A `<path>` string must use UNIX-style directory separators (forward
slash `/`), may contain any byte other than `LF`, and must not
start with double quote (`"`).
If an `LF` or double quote must be encoded into `<path>` shell-style
quoting should be used, e.g. `"path/with\n and \" in it"`.
-The value of `<path>` must be in canoncial form. That is it must not:
+The value of `<path>` must be in canonical form. That is it must not:
* contain an empty directory component (e.g. `foo//bar` is invalid),
-* end with a directory seperator (e.g. `foo/` is invalid),
-* start with a directory seperator (e.g. `/foo` is invalid),
+* end with a directory separator (e.g. `foo/` is invalid),
+* start with a directory separator (e.g. `/foo` is invalid),
* contain the special component `.` or `..` (e.g. `foo/./bar` and
`foo/../bar` are invalid).
`filedelete`
^^^^^^^^^^^^
-Included in a `commit` command to remove a file from the branch.
-If the file removal makes its directory empty, the directory will
+Included in a `commit` command to remove a file or recursively
+delete an entire directory from the branch. If the file or directory
+removal makes its parent directory empty, the parent directory will
be automatically removed too. This cascades up the tree until the
first non-empty directory or the root is reached.
'D' SP <path> LF
....
-here `<path>` is the complete path of the file to be removed.
+here `<path>` is the complete path of the file or subdirectory to
+be removed from the branch.
See `filemodify` above for a detailed description of `<path>`.
+`filecopy`
+^^^^^^^^^^^^
+Recursively copies an existing file or subdirectory to a different
+location within the branch. The existing file or directory must
+exist. If the destination exists it will be completely replaced
+by the content copied from the source.
+
+....
+ 'C' SP <path> SP <path> LF
+....
+
+here the first `<path>` is the source location and the second
+`<path>` is the destination. See `filemodify` above for a detailed
+description of what `<path>` may look like. To use a source path
+that contains SP the path must be quoted.
+
+A `filecopy` command takes effect immediately. Once the source
+location has been copied to the destination any future commands
+applied to the source location will not impact the destination of
+the copy.
+
+`filerename`
+^^^^^^^^^^^^
+Renames an existing file or subdirectory to a different location
+within the branch. The existing file or directory must exist. If
+the destination exists it will be replaced by the source directory.
+
+....
+ 'R' SP <path> SP <path> LF
+....
+
+here the first `<path>` is the source location and the second
+`<path>` is the destination. See `filemodify` above for a detailed
+description of what `<path>` may look like. To use a source path
+that contains SP the path must be quoted.
+
+A `filerename` command takes effect immediately. Once the source
+location has been renamed to the destination any future commands
+applied to the source location will create new files there and not
+impact the destination of the rename.
+
+Note that a `filerename` is the same as a `filecopy` followed by a
+`filedelete` of the source location. There is a slight performance
+advantage to using `filerename`, but the advantage is so small
+that it is never worth trying to convert a delete/add pair in
+source material into a rename for fast-import. This `filerename`
+command is provided just to simplify frontends that already have
+rename information and don't want bother with decomposing it into a
+`filecopy` followed by a `filedelete`.
+
`filedeleteall`
^^^^^^^^^^^^^^^
Included in a `commit` command to remove all files (and also all
'from' SP <committish> LF
'tagger' SP <name> SP LT <email> GT SP <when> LF
data
- LF
....
where `<name>` is the name of the tag to create.
complete set of bytes which normally goes into such a signature.
If signing is required, create lightweight tags from within fast-import with
`reset`, then create the annotated versions of those tags offline
-with the standard gitlink:git-tag[1] process.
+with the standard linkgit:git-tag[1] process.
`reset`
~~~~~~~
....
'reset' SP <ref> LF
('from' SP <committish> LF)?
- LF
+ LF?
....
For a detailed description of `<ref>` and `<committish>` see above
under `commit` and `from`.
+The `LF` after the command is optional (it used to be required).
+
The `reset` command can also be used to create lightweight
(non-annotated) tags. For example:
exact byte count format, as it is more robust and performs better.
The delimited format is intended primarily for testing fast-import.
+Comment lines appearing within the `<raw>` part of `data` commands
+are always taken to be part of the body of the data and are therefore
+never ignored by fast-import. This makes it safe to import any
+file/message content whose lines might start with `#`.
+
Exact byte count format::
The frontend must specify the number of bytes of data.
+
....
'data' SP <count> LF
- <raw> LF
+ <raw> LF?
....
+
where `<count>` is the exact number of bytes appearing within
`<raw>`. The value of `<count>` is expressed as an ASCII decimal
integer. The `LF` on either side of `<raw>` is not
included in `<count>` and will not be included in the imported data.
++
+The `LF` after `<raw>` is optional (it used to be required) but
+recommended. Always including it makes debugging a fast-import
+stream easier as the next command always starts in column 0
+of the next line, even if `<raw>` did not end with an `LF`.
Delimited format::
A delimiter string is used to mark the end of the data.
fast-import will compute the length by searching for the delimiter.
- This format is primarly useful for testing and is not
+ This format is primarily useful for testing and is not
recommended for real data.
+
....
'data' SP '<<' <delim> LF
<raw> LF
<delim> LF
+ LF?
....
+
where `<delim>` is the chosen delimiter string. The string `<delim>`
immediately trailing `<raw>` is part of `<raw>`. This is one of
the limitations of the delimited format, it is impossible to supply
a data chunk which does not have an LF as its last byte.
++
+The `LF` after `<delim> LF` is optional (it used to be required).
`checkpoint`
~~~~~~~~~~~~
....
'checkpoint' LF
- LF
+ LF?
....
Note that fast-import automatically switches packfiles when the current
repository can be loaded into Git through fast-import in about 3 hours,
explicit checkpointing may not be necessary.
+The `LF` after the command is optional (it used to be required).
+
+`progress`
+~~~~~~~~~~
+Causes fast-import to print the entire `progress` line unmodified to
+its standard output channel (file descriptor 1) when the command is
+processed from the input stream. The command otherwise has no impact
+on the current import, or on any of fast-import's internal state.
+
+....
+ 'progress' SP <any> LF
+ LF?
+....
+
+The `<any>` part of the command may contain any sequence of bytes
+that does not contain `LF`. The `LF` after the command is optional.
+Callers may wish to process the output through a tool such as sed to
+remove the leading part of the line, for example:
+
+====
+ frontend | git-fast-import | sed 's/^progress //'
+====
+
+Placing a `progress` command immediately after a `checkpoint` will
+inform the reader when the `checkpoint` has been completed and it
+can safely access the refs that fast-import updated.
+
+Crash Reports
+-------------
+If fast-import is supplied invalid input it will terminate with a
+non-zero exit status and create a crash report in the top level of
+the Git repository it was importing into. Crash reports contain
+a snapshot of the internal fast-import state as well as the most
+recent commands that lead up to the crash.
+
+All recent commands (including stream comments, file changes and
+progress commands) are shown in the command history within the crash
+report, but raw file data and commit messages are excluded from the
+crash report. This exclusion saves space within the report file
+and reduces the amount of buffering that fast-import must perform
+during execution.
+
+After writing a crash report fast-import will close the current
+packfile and export the marks table. This allows the frontend
+developer to inspect the repository state and resume the import from
+the point where it crashed. The modified branches and tags are not
+updated during a crash, as the import did not complete successfully.
+Branch and tag information can be found in the crash report and
+must be applied manually if the update is needed.
+
+An example crash:
+
+====
+ $ cat >in <<END_OF_INPUT
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ this is my commit
+ EOF
+ M 644 inline .gitignore
+ data <<EOF
+ .gitignore
+ EOF
+ M 777 inline bob
+ END_OF_INPUT
+
+ $ git-fast-import <in
+ fatal: Corrupt mode: M 777 inline bob
+ fast-import: dumping crash report to .git/fast_import_crash_8434
+
+ $ cat .git/fast_import_crash_8434
+ fast-import crash report:
+ fast-import process: 8434
+ parent process : 1391
+ at Sat Sep 1 00:58:12 2007
+
+ fatal: Corrupt mode: M 777 inline bob
+
+ Most Recent Commands Before Crash
+ ---------------------------------
+ # my very first test commit
+ commit refs/heads/master
+ committer Shawn O. Pearce <spearce> 19283 -0400
+ # who is that guy anyway?
+ data <<EOF
+ M 644 inline .gitignore
+ data <<EOF
+ * M 777 inline bob
+
+ Active Branch LRU
+ -----------------
+ active_branches = 1 cur, 5 max
+
+ pos clock name
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ 1) 0 refs/heads/master
+
+ Inactive Branches
+ -----------------
+ refs/heads/master:
+ status : active loaded dirty
+ tip commit : 0000000000000000000000000000000000000000
+ old tree : 0000000000000000000000000000000000000000
+ cur tree : 0000000000000000000000000000000000000000
+ commit clock: 0
+ last pack :
+
+
+ -------------------
+ END OF CRASH REPORT
+====
Tips and Tricks
---------------
When committing fixups, consider using `merge` to connect the
commit(s) which are supplying file revisions to the fixup branch.
-Doing so will allow tools such as gitlink:git-blame[1] to track
+Doing so will allow tools such as linkgit:git-blame[1] to track
through the real commit history and properly annotate the source
files.
Import Now, Repack Later
~~~~~~~~~~~~~~~~~~~~~~~~
As soon as fast-import completes the Git repository is completely valid
-and ready for use. Typicallly this takes only a very short time,
+and ready for use. Typically this takes only a very short time,
even for considerably large projects (100,000+ commits).
However repacking the repository is necessary to improve data
~~~~~~~~~~~~~~~~~~~~~~~~~
If you are repacking very old imported data (e.g. older than the
last year), consider expending some extra CPU time and supplying
-\--window=50 (or higher) when you run gitlink:git-repack[1].
+\--window=50 (or higher) when you run linkgit:git-repack[1].
This will take longer, but will also produce a smaller packfile.
You only need to expend the effort once, and everyone using your
project will benefit from the smaller repository.
+Include Some Progress Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Every once in a while have your frontend emit a `progress` message
+to fast-import. The contents of the messages are entirely free-form,
+so one suggestion would be to output the current month and year
+each time the current commit date moves into the next month.
+Your users will feel better knowing how much of the data stream
+has been processed.
+
Packfile Optimization
---------------------
------------------
There are a number of factors which affect how much memory fast-import
requires to perform an import. Like critical sections of core
-Git, fast-import uses its own memory allocators to ammortize any overheads
-associated with malloc. In practice fast-import tends to ammoritize any
+Git, fast-import uses its own memory allocators to amortize any overheads
+associated with malloc. In practice fast-import tends to amortize any
malloc overheads to 0, due to its use of large block allocations.
per object
~~~~~~~~~~~~~~~
Trees (aka directories) use just 12 bytes of memory on top of the
memory required for their entries (see ``per active file'' below).
-The cost of a tree is virtually 0, as its overhead ammortizes out
+The cost of a tree is virtually 0, as its overhead amortizes out
over the individual file entries.
per active file entry
GIT
---
-Part of the gitlink:git[7] suite
-
+Part of the linkgit:git[7] suite