Documentation / git-fast-import.txton commit revision.c: introduce --min-parents and --max-parents options (ad5aeed)
   1git-fast-import(1)
   2==================
   3
   4NAME
   5----
   6git-fast-import - Backend for fast Git data importers
   7
   8
   9SYNOPSIS
  10--------
  11frontend | 'git fast-import' [options]
  12
  13DESCRIPTION
  14-----------
  15This program is usually not what the end user wants to run directly.
  16Most end users want to use one of the existing frontend programs,
  17which parses a specific type of foreign source and feeds the contents
  18stored there to 'git fast-import'.
  19
  20fast-import reads a mixed command/data stream from standard input and
  21writes one or more packfiles directly into the current repository.
  22When EOF is received on standard input, fast import writes out
  23updated branch and tag refs, fully updating the current repository
  24with the newly imported data.
  25
  26The fast-import backend itself can import into an empty repository (one that
  27has already been initialized by 'git init') or incrementally
  28update an existing populated repository.  Whether or not incremental
  29imports are supported from a particular foreign source depends on
  30the frontend program in use.
  31
  32
  33OPTIONS
  34-------
  35--date-format=<fmt>::
  36        Specify the type of dates the frontend will supply to
  37        fast-import within `author`, `committer` and `tagger` commands.
  38        See ``Date Formats'' below for details about which formats
  39        are supported, and their syntax.
  40
  41--force::
  42        Force updating modified existing branches, even if doing
  43        so would cause commits to be lost (as the new commit does
  44        not contain the old commit).
  45
  46--max-pack-size=<n>::
  47        Maximum size of each output packfile.
  48        The default is unlimited.
  49
  50--big-file-threshold=<n>::
  51        Maximum size of a blob that fast-import will attempt to
  52        create a delta for, expressed in bytes.  The default is 512m
  53        (512 MiB).  Some importers may wish to lower this on systems
  54        with constrained memory.
  55
  56--depth=<n>::
  57        Maximum delta depth, for blob and tree deltification.
  58        Default is 10.
  59
  60--active-branches=<n>::
  61        Maximum number of branches to maintain active at once.
  62        See ``Memory Utilization'' below for details.  Default is 5.
  63
  64--export-marks=<file>::
  65        Dumps the internal marks table to <file> when complete.
  66        Marks are written one per line as `:markid SHA-1`.
  67        Frontends can use this file to validate imports after they
  68        have been completed, or to save the marks table across
  69        incremental runs.  As <file> is only opened and truncated
  70        at checkpoint (or completion) the same path can also be
  71        safely given to \--import-marks.
  72
  73--import-marks=<file>::
  74        Before processing any input, load the marks specified in
  75        <file>.  The input file must exist, must be readable, and
  76        must use the same format as produced by \--export-marks.
  77        Multiple options may be supplied to import more than one
  78        set of marks.  If a mark is defined to different values,
  79        the last file wins.
  80
  81--relative-marks::
  82        After specifying --relative-marks= the paths specified
  83        with --import-marks= and --export-marks= are relative
  84        to an internal directory in the current repository.
  85        In git-fast-import this means that the paths are relative
  86        to the .git/info/fast-import directory. However, other
  87        importers may use a different location.
  88
  89--no-relative-marks::
  90        Negates a previous --relative-marks. Allows for combining
  91        relative and non-relative marks by interweaving
  92        --(no-)-relative-marks= with the --(import|export)-marks=
  93        options.
  94
  95--cat-blob-fd=<fd>::
  96        Specify the file descriptor that will be written to
  97        when the `cat-blob` command is encountered in the stream.
  98        The default behaviour is to write to `stdout`.
  99
 100--export-pack-edges=<file>::
 101        After creating a packfile, print a line of data to
 102        <file> listing the filename of the packfile and the last
 103        commit on each branch that was written to that packfile.
 104        This information may be useful after importing projects
 105        whose total object set exceeds the 4 GiB packfile limit,
 106        as these commits can be used as edge points during calls
 107        to 'git pack-objects'.
 108
 109--quiet::
 110        Disable all non-fatal output, making fast-import silent when it
 111        is successful.  This option disables the output shown by
 112        \--stats.
 113
 114--stats::
 115        Display some basic statistics about the objects fast-import has
 116        created, the packfiles they were stored into, and the
 117        memory used by fast-import during this run.  Showing this output
 118        is currently the default, but can be disabled with \--quiet.
 119
 120
 121Performance
 122-----------
 123The design of fast-import allows it to import large projects in a minimum
 124amount of memory usage and processing time.  Assuming the frontend
 125is able to keep up with fast-import and feed it a constant stream of data,
 126import times for projects holding 10+ years of history and containing
 127100,000+ individual commits are generally completed in just 1-2
 128hours on quite modest (~$2,000 USD) hardware.
 129
 130Most bottlenecks appear to be in foreign source data access (the
 131source just cannot extract revisions fast enough) or disk IO (fast-import
 132writes as fast as the disk will take the data).  Imports will run
 133faster if the source data is stored on a different drive than the
 134destination Git repository (due to less IO contention).
 135
 136
 137Development Cost
 138----------------
 139A typical frontend for fast-import tends to weigh in at approximately 200
 140lines of Perl/Python/Ruby code.  Most developers have been able to
 141create working importers in just a couple of hours, even though it
 142is their first exposure to fast-import, and sometimes even to Git.  This is
 143an ideal situation, given that most conversion tools are throw-away
 144(use once, and never look back).
 145
 146
 147Parallel Operation
 148------------------
 149Like 'git push' or 'git fetch', imports handled by fast-import are safe to
 150run alongside parallel `git repack -a -d` or `git gc` invocations,
 151or any other Git operation (including 'git prune', as loose objects
 152are never used by fast-import).
 153
 154fast-import does not lock the branch or tag refs it is actively importing.
 155After the import, during its ref update phase, fast-import tests each
 156existing branch ref to verify the update will be a fast-forward
 157update (the commit stored in the ref is contained in the new
 158history of the commit to be written).  If the update is not a
 159fast-forward update, fast-import will skip updating that ref and instead
 160prints a warning message.  fast-import will always attempt to update all
 161branch refs, and does not stop on the first failure.
 162
 163Branch updates can be forced with \--force, but it's recommended that
 164this only be used on an otherwise quiet repository.  Using \--force
 165is not necessary for an initial import into an empty repository.
 166
 167
 168Technical Discussion
 169--------------------
 170fast-import tracks a set of branches in memory.  Any branch can be created
 171or modified at any point during the import process by sending a
 172`commit` command on the input stream.  This design allows a frontend
 173program to process an unlimited number of branches simultaneously,
 174generating commits in the order they are available from the source
 175data.  It also simplifies the frontend programs considerably.
 176
 177fast-import does not use or alter the current working directory, or any
 178file within it.  (It does however update the current Git repository,
 179as referenced by `GIT_DIR`.)  Therefore an import frontend may use
 180the working directory for its own purposes, such as extracting file
 181revisions from the foreign source.  This ignorance of the working
 182directory also allows fast-import to run very quickly, as it does not
 183need to perform any costly file update operations when switching
 184between branches.
 185
 186Input Format
 187------------
 188With the exception of raw file data (which Git does not interpret)
 189the fast-import input format is text (ASCII) based.  This text based
 190format simplifies development and debugging of frontend programs,
 191especially when a higher level language such as Perl, Python or
 192Ruby is being used.
 193
 194fast-import is very strict about its input.  Where we say SP below we mean
 195*exactly* one space.  Likewise LF means one (and only one) linefeed.
 196Supplying additional whitespace characters will cause unexpected
 197results, such as branch names or file names with leading or trailing
 198spaces in their name, or early termination of fast-import when it encounters
 199unexpected input.
 200
 201Stream Comments
 202~~~~~~~~~~~~~~~
 203To aid in debugging frontends fast-import ignores any line that
 204begins with `#` (ASCII pound/hash) up to and including the line
 205ending `LF`.  A comment line may contain any sequence of bytes
 206that does not contain an LF and therefore may be used to include
 207any detailed debugging information that might be specific to the
 208frontend and useful when inspecting a fast-import data stream.
 209
 210Date Formats
 211~~~~~~~~~~~~
 212The following date formats are supported.  A frontend should select
 213the format it will use for this import by passing the format name
 214in the \--date-format=<fmt> command line option.
 215
 216`raw`::
 217        This is the Git native format and is `<time> SP <offutc>`.
 218        It is also fast-import's default format, if \--date-format was
 219        not specified.
 220+
 221The time of the event is specified by `<time>` as the number of
 222seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
 223written as an ASCII decimal integer.
 224+
 225The local offset is specified by `<offutc>` as a positive or negative
 226offset from UTC.  For example EST (which is 5 hours behind UTC)
 227would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''.
 228The local offset does not affect `<time>`; it is used only as an
 229advisement to help formatting routines display the timestamp.
 230+
 231If the local offset is not available in the source material, use
 232``+0000'', or the most common local offset.  For example many
 233organizations have a CVS repository which has only ever been accessed
 234by users who are located in the same location and timezone.  In this
 235case a reasonable offset from UTC could be assumed.
 236+
 237Unlike the `rfc2822` format, this format is very strict.  Any
 238variation in formatting will cause fast-import to reject the value.
 239
 240`rfc2822`::
 241        This is the standard email format as described by RFC 2822.
 242+
 243An example value is ``Tue Feb 6 11:22:18 2007 -0500''.  The Git
 244parser is accurate, but a little on the lenient side.  It is the
 245same parser used by 'git am' when applying patches
 246received from email.
 247+
 248Some malformed strings may be accepted as valid dates.  In some of
 249these cases Git will still be able to obtain the correct date from
 250the malformed string.  There are also some types of malformed
 251strings which Git will parse wrong, and yet consider valid.
 252Seriously malformed strings will be rejected.
 253+
 254Unlike the `raw` format above, the timezone/UTC offset information
 255contained in an RFC 2822 date string is used to adjust the date
 256value to UTC prior to storage.  Therefore it is important that
 257this information be as accurate as possible.
 258+
 259If the source material uses RFC 2822 style dates,
 260the frontend should let fast-import handle the parsing and conversion
 261(rather than attempting to do it itself) as the Git parser has
 262been well tested in the wild.
 263+
 264Frontends should prefer the `raw` format if the source material
 265already uses UNIX-epoch format, can be coaxed to give dates in that
 266format, or its format is easily convertible to it, as there is no
 267ambiguity in parsing.
 268
 269`now`::
 270        Always use the current time and timezone.  The literal
 271        `now` must always be supplied for `<when>`.
 272+
 273This is a toy format.  The current time and timezone of this system
 274is always copied into the identity string at the time it is being
 275created by fast-import.  There is no way to specify a different time or
 276timezone.
 277+
 278This particular format is supplied as it's short to implement and
 279may be useful to a process that wants to create a new commit
 280right now, without needing to use a working directory or
 281'git update-index'.
 282+
 283If separate `author` and `committer` commands are used in a `commit`
 284the timestamps may not match, as the system clock will be polled
 285twice (once for each command).  The only way to ensure that both
 286author and committer identity information has the same timestamp
 287is to omit `author` (thus copying from `committer`) or to use a
 288date format other than `now`.
 289
 290Commands
 291~~~~~~~~
 292fast-import accepts several commands to update the current repository
 293and control the current import process.  More detailed discussion
 294(with examples) of each command follows later.
 295
 296`commit`::
 297        Creates a new branch or updates an existing branch by
 298        creating a new commit and updating the branch to point at
 299        the newly created commit.
 300
 301`tag`::
 302        Creates an annotated tag object from an existing commit or
 303        branch.  Lightweight tags are not supported by this command,
 304        as they are not recommended for recording meaningful points
 305        in time.
 306
 307`reset`::
 308        Reset an existing branch (or a new branch) to a specific
 309        revision.  This command must be used to change a branch to
 310        a specific revision without making a commit on it.
 311
 312`blob`::
 313        Convert raw file data into a blob, for future use in a
 314        `commit` command.  This command is optional and is not
 315        needed to perform an import.
 316
 317`checkpoint`::
 318        Forces fast-import to close the current packfile, generate its
 319        unique SHA-1 checksum and index, and start a new packfile.
 320        This command is optional and is not needed to perform
 321        an import.
 322
 323`progress`::
 324        Causes fast-import to echo the entire line to its own
 325        standard output.  This command is optional and is not needed
 326        to perform an import.
 327
 328`cat-blob`::
 329        Causes fast-import to print a blob in 'cat-file --batch'
 330        format to the file descriptor set with `--cat-blob-fd` or
 331        `stdout` if unspecified.
 332
 333`feature`::
 334        Require that fast-import supports the specified feature, or
 335        abort if it does not.
 336
 337`option`::
 338        Specify any of the options listed under OPTIONS that do not
 339        change stream semantic to suit the frontend's needs. This
 340        command is optional and is not needed to perform an import.
 341
 342`commit`
 343~~~~~~~~
 344Create or update a branch with a new commit, recording one logical
 345change to the project.
 346
 347....
 348        'commit' SP <ref> LF
 349        mark?
 350        ('author' (SP <name>)? SP LT <email> GT SP <when> LF)?
 351        'committer' (SP <name>)? SP LT <email> GT SP <when> LF
 352        data
 353        ('from' SP <committish> LF)?
 354        ('merge' SP <committish> LF)?
 355        (filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*
 356        LF?
 357....
 358
 359where `<ref>` is the name of the branch to make the commit on.
 360Typically branch names are prefixed with `refs/heads/` in
 361Git, so importing the CVS branch symbol `RELENG-1_0` would use
 362`refs/heads/RELENG-1_0` for the value of `<ref>`.  The value of
 363`<ref>` must be a valid refname in Git.  As `LF` is not valid in
 364a Git refname, no quoting or escaping syntax is supported here.
 365
 366A `mark` command may optionally appear, requesting fast-import to save a
 367reference to the newly created commit for future use by the frontend
 368(see below for format).  It is very common for frontends to mark
 369every commit they create, thereby allowing future branch creation
 370from any imported commit.
 371
 372The `data` command following `committer` must supply the commit
 373message (see below for `data` command syntax).  To import an empty
 374commit message use a 0 length data.  Commit messages are free-form
 375and are not interpreted by Git.  Currently they must be encoded in
 376UTF-8, as fast-import does not permit other encodings to be specified.
 377
 378Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`,
 379`filedeleteall` and `notemodify` commands
 380may be included to update the contents of the branch prior to
 381creating the commit.  These commands may be supplied in any order.
 382However it is recommended that a `filedeleteall` command precede
 383all `filemodify`, `filecopy`, `filerename` and `notemodify` commands in
 384the same commit, as `filedeleteall` wipes the branch clean (see below).
 385
 386The `LF` after the command is optional (it used to be required).
 387
 388`author`
 389^^^^^^^^
 390An `author` command may optionally appear, if the author information
 391might differ from the committer information.  If `author` is omitted
 392then fast-import will automatically use the committer's information for
 393the author portion of the commit.  See below for a description of
 394the fields in `author`, as they are identical to `committer`.
 395
 396`committer`
 397^^^^^^^^^^^
 398The `committer` command indicates who made this commit, and when
 399they made it.
 400
 401Here `<name>` is the person's display name (for example
 402``Com M Itter'') and `<email>` is the person's email address
 403(``cm@example.com'').  `LT` and `GT` are the literal less-than (\x3c)
 404and greater-than (\x3e) symbols.  These are required to delimit
 405the email address from the other fields in the line.  Note that
 406`<name>` is free-form and may contain any sequence of bytes, except
 407`LT` and `LF`.  It is typically UTF-8 encoded.
 408
 409The time of the change is specified by `<when>` using the date format
 410that was selected by the \--date-format=<fmt> command line option.
 411See ``Date Formats'' above for the set of supported formats, and
 412their syntax.
 413
 414`from`
 415^^^^^^
 416The `from` command is used to specify the commit to initialize
 417this branch from.  This revision will be the first ancestor of the
 418new commit.
 419
 420Omitting the `from` command in the first commit of a new branch
 421will cause fast-import to create that commit with no ancestor. This
 422tends to be desired only for the initial commit of a project.
 423If the frontend creates all files from scratch when making a new
 424branch, a `merge` command may be used instead of `from` to start
 425the commit with an empty tree.
 426Omitting the `from` command on existing branches is usually desired,
 427as the current commit on that branch is automatically assumed to
 428be the first ancestor of the new commit.
 429
 430As `LF` is not valid in a Git refname or SHA-1 expression, no
 431quoting or escaping syntax is supported within `<committish>`.
 432
 433Here `<committish>` is any of the following:
 434
 435* The name of an existing branch already in fast-import's internal branch
 436  table.  If fast-import doesn't know the name, it's treated as a SHA-1
 437  expression.
 438
 439* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
 440+
 441The reason fast-import uses `:` to denote a mark reference is this character
 442is not legal in a Git branch name.  The leading `:` makes it easy
 443to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
 444or `refs/heads/42`), or an abbreviated SHA-1 which happened to
 445consist only of base-10 digits.
 446+
 447Marks must be declared (via `mark`) before they can be used.
 448
 449* A complete 40 byte or abbreviated commit SHA-1 in hex.
 450
 451* Any valid Git SHA-1 expression that resolves to a commit.  See
 452  ``SPECIFYING REVISIONS'' in linkgit:gitrevisions[7] for details.
 453
 454The special case of restarting an incremental import from the
 455current branch value should be written as:
 456----
 457        from refs/heads/branch^0
 458----
 459The `{caret}0` suffix is necessary as fast-import does not permit a branch to
 460start from itself, and the branch is created in memory before the
 461`from` command is even read from the input.  Adding `{caret}0` will force
 462fast-import to resolve the commit through Git's revision parsing library,
 463rather than its internal branch table, thereby loading in the
 464existing value of the branch.
 465
 466`merge`
 467^^^^^^^
 468Includes one additional ancestor commit.  If the `from` command is
 469omitted when creating a new branch, the first `merge` commit will be
 470the first ancestor of the current commit, and the branch will start
 471out with no files.  An unlimited number of `merge` commands per
 472commit are permitted by fast-import, thereby establishing an n-way merge.
 473However Git's other tools never create commits with more than 15
 474additional ancestors (forming a 16-way merge).  For this reason
 475it is suggested that frontends do not use more than 15 `merge`
 476commands per commit; 16, if starting a new, empty branch.
 477
 478Here `<committish>` is any of the commit specification expressions
 479also accepted by `from` (see above).
 480
 481`filemodify`
 482^^^^^^^^^^^^
 483Included in a `commit` command to add a new file or change the
 484content of an existing file.  This command has two different means
 485of specifying the content of the file.
 486
 487External data format::
 488        The data content for the file was already supplied by a prior
 489        `blob` command.  The frontend just needs to connect it.
 490+
 491....
 492        'M' SP <mode> SP <dataref> SP <path> LF
 493....
 494+
 495Here usually `<dataref>` must be either a mark reference (`:<idnum>`)
 496set by a prior `blob` command, or a full 40-byte SHA-1 of an
 497existing Git blob object.  If `<mode>` is `040000`` then
 498`<dataref>` must be the full 40-byte SHA-1 of an existing
 499Git tree object or a mark reference set with `--import-marks`.
 500
 501Inline data format::
 502        The data content for the file has not been supplied yet.
 503        The frontend wants to supply it as part of this modify
 504        command.
 505+
 506....
 507        'M' SP <mode> SP 'inline' SP <path> LF
 508        data
 509....
 510+
 511See below for a detailed description of the `data` command.
 512
 513In both formats `<mode>` is the type of file entry, specified
 514in octal.  Git only supports the following modes:
 515
 516* `100644` or `644`: A normal (not-executable) file.  The majority
 517  of files in most projects use this mode.  If in doubt, this is
 518  what you want.
 519* `100755` or `755`: A normal, but executable, file.
 520* `120000`: A symlink, the content of the file will be the link target.
 521* `160000`: A gitlink, SHA-1 of the object refers to a commit in
 522  another repository. Git links can only be specified by SHA or through
 523  a commit mark. They are used to implement submodules.
 524* `040000`: A subdirectory.  Subdirectories can only be specified by
 525  SHA or through a tree mark set with `--import-marks`.
 526
 527In both formats `<path>` is the complete path of the file to be added
 528(if not already existing) or modified (if already existing).
 529
 530A `<path>` string must use UNIX-style directory separators (forward
 531slash `/`), may contain any byte other than `LF`, and must not
 532start with double quote (`"`).
 533
 534If an `LF` or double quote must be encoded into `<path>` shell-style
 535quoting should be used, e.g. `"path/with\n and \" in it"`.
 536
 537The value of `<path>` must be in canonical form. That is it must not:
 538
 539* contain an empty directory component (e.g. `foo//bar` is invalid),
 540* end with a directory separator (e.g. `foo/` is invalid),
 541* start with a directory separator (e.g. `/foo` is invalid),
 542* contain the special component `.` or `..` (e.g. `foo/./bar` and
 543  `foo/../bar` are invalid).
 544
 545The root of the tree can be represented by an empty string as `<path>`.
 546
 547It is recommended that `<path>` always be encoded using UTF-8.
 548
 549`filedelete`
 550^^^^^^^^^^^^
 551Included in a `commit` command to remove a file or recursively
 552delete an entire directory from the branch.  If the file or directory
 553removal makes its parent directory empty, the parent directory will
 554be automatically removed too.  This cascades up the tree until the
 555first non-empty directory or the root is reached.
 556
 557....
 558        'D' SP <path> LF
 559....
 560
 561here `<path>` is the complete path of the file or subdirectory to
 562be removed from the branch.
 563See `filemodify` above for a detailed description of `<path>`.
 564
 565`filecopy`
 566^^^^^^^^^^^^
 567Recursively copies an existing file or subdirectory to a different
 568location within the branch.  The existing file or directory must
 569exist.  If the destination exists it will be completely replaced
 570by the content copied from the source.
 571
 572....
 573        'C' SP <path> SP <path> LF
 574....
 575
 576here the first `<path>` is the source location and the second
 577`<path>` is the destination.  See `filemodify` above for a detailed
 578description of what `<path>` may look like.  To use a source path
 579that contains SP the path must be quoted.
 580
 581A `filecopy` command takes effect immediately.  Once the source
 582location has been copied to the destination any future commands
 583applied to the source location will not impact the destination of
 584the copy.
 585
 586`filerename`
 587^^^^^^^^^^^^
 588Renames an existing file or subdirectory to a different location
 589within the branch.  The existing file or directory must exist. If
 590the destination exists it will be replaced by the source directory.
 591
 592....
 593        'R' SP <path> SP <path> LF
 594....
 595
 596here the first `<path>` is the source location and the second
 597`<path>` is the destination.  See `filemodify` above for a detailed
 598description of what `<path>` may look like.  To use a source path
 599that contains SP the path must be quoted.
 600
 601A `filerename` command takes effect immediately.  Once the source
 602location has been renamed to the destination any future commands
 603applied to the source location will create new files there and not
 604impact the destination of the rename.
 605
 606Note that a `filerename` is the same as a `filecopy` followed by a
 607`filedelete` of the source location.  There is a slight performance
 608advantage to using `filerename`, but the advantage is so small
 609that it is never worth trying to convert a delete/add pair in
 610source material into a rename for fast-import.  This `filerename`
 611command is provided just to simplify frontends that already have
 612rename information and don't want bother with decomposing it into a
 613`filecopy` followed by a `filedelete`.
 614
 615`filedeleteall`
 616^^^^^^^^^^^^^^^
 617Included in a `commit` command to remove all files (and also all
 618directories) from the branch.  This command resets the internal
 619branch structure to have no files in it, allowing the frontend
 620to subsequently add all interesting files from scratch.
 621
 622....
 623        'deleteall' LF
 624....
 625
 626This command is extremely useful if the frontend does not know
 627(or does not care to know) what files are currently on the branch,
 628and therefore cannot generate the proper `filedelete` commands to
 629update the content.
 630
 631Issuing a `filedeleteall` followed by the needed `filemodify`
 632commands to set the correct content will produce the same results
 633as sending only the needed `filemodify` and `filedelete` commands.
 634The `filedeleteall` approach may however require fast-import to use slightly
 635more memory per active branch (less than 1 MiB for even most large
 636projects); so frontends that can easily obtain only the affected
 637paths for a commit are encouraged to do so.
 638
 639`notemodify`
 640^^^^^^^^^^^^
 641Included in a `commit` command to add a new note (annotating a given
 642commit) or change the content of an existing note.  This command has
 643two different means of specifying the content of the note.
 644
 645External data format::
 646        The data content for the note was already supplied by a prior
 647        `blob` command.  The frontend just needs to connect it to the
 648        commit that is to be annotated.
 649+
 650....
 651        'N' SP <dataref> SP <committish> LF
 652....
 653+
 654Here `<dataref>` can be either a mark reference (`:<idnum>`)
 655set by a prior `blob` command, or a full 40-byte SHA-1 of an
 656existing Git blob object.
 657
 658Inline data format::
 659        The data content for the note has not been supplied yet.
 660        The frontend wants to supply it as part of this modify
 661        command.
 662+
 663....
 664        'N' SP 'inline' SP <committish> LF
 665        data
 666....
 667+
 668See below for a detailed description of the `data` command.
 669
 670In both formats `<committish>` is any of the commit specification
 671expressions also accepted by `from` (see above).
 672
 673`mark`
 674~~~~~~
 675Arranges for fast-import to save a reference to the current object, allowing
 676the frontend to recall this object at a future point in time, without
 677knowing its SHA-1.  Here the current object is the object creation
 678command the `mark` command appears within.  This can be `commit`,
 679`tag`, and `blob`, but `commit` is the most common usage.
 680
 681....
 682        'mark' SP ':' <idnum> LF
 683....
 684
 685where `<idnum>` is the number assigned by the frontend to this mark.
 686The value of `<idnum>` is expressed as an ASCII decimal integer.
 687The value 0 is reserved and cannot be used as
 688a mark.  Only values greater than or equal to 1 may be used as marks.
 689
 690New marks are created automatically.  Existing marks can be moved
 691to another object simply by reusing the same `<idnum>` in another
 692`mark` command.
 693
 694`tag`
 695~~~~~
 696Creates an annotated tag referring to a specific commit.  To create
 697lightweight (non-annotated) tags see the `reset` command below.
 698
 699....
 700        'tag' SP <name> LF
 701        'from' SP <committish> LF
 702        'tagger' (SP <name>)? SP LT <email> GT SP <when> LF
 703        data
 704....
 705
 706where `<name>` is the name of the tag to create.
 707
 708Tag names are automatically prefixed with `refs/tags/` when stored
 709in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
 710use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the
 711corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
 712
 713The value of `<name>` must be a valid refname in Git and therefore
 714may contain forward slashes.  As `LF` is not valid in a Git refname,
 715no quoting or escaping syntax is supported here.
 716
 717The `from` command is the same as in the `commit` command; see
 718above for details.
 719
 720The `tagger` command uses the same format as `committer` within
 721`commit`; again see above for details.
 722
 723The `data` command following `tagger` must supply the annotated tag
 724message (see below for `data` command syntax).  To import an empty
 725tag message use a 0 length data.  Tag messages are free-form and are
 726not interpreted by Git.  Currently they must be encoded in UTF-8,
 727as fast-import does not permit other encodings to be specified.
 728
 729Signing annotated tags during import from within fast-import is not
 730supported.  Trying to include your own PGP/GPG signature is not
 731recommended, as the frontend does not (easily) have access to the
 732complete set of bytes which normally goes into such a signature.
 733If signing is required, create lightweight tags from within fast-import with
 734`reset`, then create the annotated versions of those tags offline
 735with the standard 'git tag' process.
 736
 737`reset`
 738~~~~~~~
 739Creates (or recreates) the named branch, optionally starting from
 740a specific revision.  The reset command allows a frontend to issue
 741a new `from` command for an existing branch, or to create a new
 742branch from an existing commit without creating a new commit.
 743
 744....
 745        'reset' SP <ref> LF
 746        ('from' SP <committish> LF)?
 747        LF?
 748....
 749
 750For a detailed description of `<ref>` and `<committish>` see above
 751under `commit` and `from`.
 752
 753The `LF` after the command is optional (it used to be required).
 754
 755The `reset` command can also be used to create lightweight
 756(non-annotated) tags.  For example:
 757
 758====
 759        reset refs/tags/938
 760        from :938
 761====
 762
 763would create the lightweight tag `refs/tags/938` referring to
 764whatever commit mark `:938` references.
 765
 766`blob`
 767~~~~~~
 768Requests writing one file revision to the packfile.  The revision
 769is not connected to any commit; this connection must be formed in
 770a subsequent `commit` command by referencing the blob through an
 771assigned mark.
 772
 773....
 774        'blob' LF
 775        mark?
 776        data
 777....
 778
 779The mark command is optional here as some frontends have chosen
 780to generate the Git SHA-1 for the blob on their own, and feed that
 781directly to `commit`.  This is typically more work than it's worth
 782however, as marks are inexpensive to store and easy to use.
 783
 784`data`
 785~~~~~~
 786Supplies raw data (for use as blob/file content, commit messages, or
 787annotated tag messages) to fast-import.  Data can be supplied using an exact
 788byte count or delimited with a terminating line.  Real frontends
 789intended for production-quality conversions should always use the
 790exact byte count format, as it is more robust and performs better.
 791The delimited format is intended primarily for testing fast-import.
 792
 793Comment lines appearing within the `<raw>` part of `data` commands
 794are always taken to be part of the body of the data and are therefore
 795never ignored by fast-import.  This makes it safe to import any
 796file/message content whose lines might start with `#`.
 797
 798Exact byte count format::
 799        The frontend must specify the number of bytes of data.
 800+
 801....
 802        'data' SP <count> LF
 803        <raw> LF?
 804....
 805+
 806where `<count>` is the exact number of bytes appearing within
 807`<raw>`.  The value of `<count>` is expressed as an ASCII decimal
 808integer.  The `LF` on either side of `<raw>` is not
 809included in `<count>` and will not be included in the imported data.
 810+
 811The `LF` after `<raw>` is optional (it used to be required) but
 812recommended.  Always including it makes debugging a fast-import
 813stream easier as the next command always starts in column 0
 814of the next line, even if `<raw>` did not end with an `LF`.
 815
 816Delimited format::
 817        A delimiter string is used to mark the end of the data.
 818        fast-import will compute the length by searching for the delimiter.
 819        This format is primarily useful for testing and is not
 820        recommended for real data.
 821+
 822....
 823        'data' SP '<<' <delim> LF
 824        <raw> LF
 825        <delim> LF
 826        LF?
 827....
 828+
 829where `<delim>` is the chosen delimiter string.  The string `<delim>`
 830must not appear on a line by itself within `<raw>`, as otherwise
 831fast-import will think the data ends earlier than it really does.  The `LF`
 832immediately trailing `<raw>` is part of `<raw>`.  This is one of
 833the limitations of the delimited format, it is impossible to supply
 834a data chunk which does not have an LF as its last byte.
 835+
 836The `LF` after `<delim> LF` is optional (it used to be required).
 837
 838`checkpoint`
 839~~~~~~~~~~~~
 840Forces fast-import to close the current packfile, start a new one, and to
 841save out all current branch refs, tags and marks.
 842
 843....
 844        'checkpoint' LF
 845        LF?
 846....
 847
 848Note that fast-import automatically switches packfiles when the current
 849packfile reaches \--max-pack-size, or 4 GiB, whichever limit is
 850smaller.  During an automatic packfile switch fast-import does not update
 851the branch refs, tags or marks.
 852
 853As a `checkpoint` can require a significant amount of CPU time and
 854disk IO (to compute the overall pack SHA-1 checksum, generate the
 855corresponding index file, and update the refs) it can easily take
 856several minutes for a single `checkpoint` command to complete.
 857
 858Frontends may choose to issue checkpoints during extremely large
 859and long running imports, or when they need to allow another Git
 860process access to a branch.  However given that a 30 GiB Subversion
 861repository can be loaded into Git through fast-import in about 3 hours,
 862explicit checkpointing may not be necessary.
 863
 864The `LF` after the command is optional (it used to be required).
 865
 866`progress`
 867~~~~~~~~~~
 868Causes fast-import to print the entire `progress` line unmodified to
 869its standard output channel (file descriptor 1) when the command is
 870processed from the input stream.  The command otherwise has no impact
 871on the current import, or on any of fast-import's internal state.
 872
 873....
 874        'progress' SP <any> LF
 875        LF?
 876....
 877
 878The `<any>` part of the command may contain any sequence of bytes
 879that does not contain `LF`.  The `LF` after the command is optional.
 880Callers may wish to process the output through a tool such as sed to
 881remove the leading part of the line, for example:
 882
 883====
 884        frontend | git fast-import | sed 's/^progress //'
 885====
 886
 887Placing a `progress` command immediately after a `checkpoint` will
 888inform the reader when the `checkpoint` has been completed and it
 889can safely access the refs that fast-import updated.
 890
 891`cat-blob`
 892~~~~~~~~~~
 893Causes fast-import to print a blob to a file descriptor previously
 894arranged with the `--cat-blob-fd` argument.  The command otherwise
 895has no impact on the current import; its main purpose is to
 896retrieve blobs that may be in fast-import's memory but not
 897accessible from the target repository.
 898
 899....
 900        'cat-blob' SP <dataref> LF
 901....
 902
 903The `<dataref>` can be either a mark reference (`:<idnum>`)
 904set previously or a full 40-byte SHA-1 of a Git blob, preexisting or
 905ready to be written.
 906
 907Output uses the same format as `git cat-file --batch`:
 908
 909====
 910        <sha1> SP 'blob' SP <size> LF
 911        <contents> LF
 912====
 913
 914This command can be used anywhere in the stream that comments are
 915accepted.  In particular, the `cat-blob` command can be used in the
 916middle of a commit but not in the middle of a `data` command.
 917
 918`feature`
 919~~~~~~~~~
 920Require that fast-import supports the specified feature, or abort if
 921it does not.
 922
 923....
 924        'feature' SP <feature> ('=' <argument>)? LF
 925....
 926
 927The <feature> part of the command may be any one of the following:
 928
 929date-format::
 930export-marks::
 931relative-marks::
 932no-relative-marks::
 933force::
 934        Act as though the corresponding command-line option with
 935        a leading '--' was passed on the command line
 936        (see OPTIONS, above).
 937
 938import-marks::
 939        Like --import-marks except in two respects: first, only one
 940        "feature import-marks" command is allowed per stream;
 941        second, an --import-marks= command-line option overrides
 942        any "feature import-marks" command in the stream.
 943
 944cat-blob::
 945        Ignored.  Versions of fast-import not supporting the
 946        "cat-blob" command will exit with a message indicating so.
 947        This lets the import error out early with a clear message,
 948        rather than wasting time on the early part of an import
 949        before the unsupported command is detected.
 950
 951`option`
 952~~~~~~~~
 953Processes the specified option so that git fast-import behaves in a
 954way that suits the frontend's needs.
 955Note that options specified by the frontend are overridden by any
 956options the user may specify to git fast-import itself.
 957
 958....
 959    'option' SP <option> LF
 960....
 961
 962The `<option>` part of the command may contain any of the options
 963listed in the OPTIONS section that do not change import semantics,
 964without the leading '--' and is treated in the same way.
 965
 966Option commands must be the first commands on the input (not counting
 967feature commands), to give an option command after any non-option
 968command is an error.
 969
 970The following commandline options change import semantics and may therefore
 971not be passed as option:
 972
 973* date-format
 974* import-marks
 975* export-marks
 976* cat-blob-fd
 977* force
 978
 979Crash Reports
 980-------------
 981If fast-import is supplied invalid input it will terminate with a
 982non-zero exit status and create a crash report in the top level of
 983the Git repository it was importing into.  Crash reports contain
 984a snapshot of the internal fast-import state as well as the most
 985recent commands that lead up to the crash.
 986
 987All recent commands (including stream comments, file changes and
 988progress commands) are shown in the command history within the crash
 989report, but raw file data and commit messages are excluded from the
 990crash report.  This exclusion saves space within the report file
 991and reduces the amount of buffering that fast-import must perform
 992during execution.
 993
 994After writing a crash report fast-import will close the current
 995packfile and export the marks table.  This allows the frontend
 996developer to inspect the repository state and resume the import from
 997the point where it crashed.  The modified branches and tags are not
 998updated during a crash, as the import did not complete successfully.
 999Branch and tag information can be found in the crash report and
1000must be applied manually if the update is needed.
1001
1002An example crash:
1003
1004====
1005        $ cat >in <<END_OF_INPUT
1006        # my very first test commit
1007        commit refs/heads/master
1008        committer Shawn O. Pearce <spearce> 19283 -0400
1009        # who is that guy anyway?
1010        data <<EOF
1011        this is my commit
1012        EOF
1013        M 644 inline .gitignore
1014        data <<EOF
1015        .gitignore
1016        EOF
1017        M 777 inline bob
1018        END_OF_INPUT
1019
1020        $ git fast-import <in
1021        fatal: Corrupt mode: M 777 inline bob
1022        fast-import: dumping crash report to .git/fast_import_crash_8434
1023
1024        $ cat .git/fast_import_crash_8434
1025        fast-import crash report:
1026            fast-import process: 8434
1027            parent process     : 1391
1028            at Sat Sep 1 00:58:12 2007
1029
1030        fatal: Corrupt mode: M 777 inline bob
1031
1032        Most Recent Commands Before Crash
1033        ---------------------------------
1034          # my very first test commit
1035          commit refs/heads/master
1036          committer Shawn O. Pearce <spearce> 19283 -0400
1037          # who is that guy anyway?
1038          data <<EOF
1039          M 644 inline .gitignore
1040          data <<EOF
1041        * M 777 inline bob
1042
1043        Active Branch LRU
1044        -----------------
1045            active_branches = 1 cur, 5 max
1046
1047          pos  clock name
1048          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1049           1)      0 refs/heads/master
1050
1051        Inactive Branches
1052        -----------------
1053        refs/heads/master:
1054          status      : active loaded dirty
1055          tip commit  : 0000000000000000000000000000000000000000
1056          old tree    : 0000000000000000000000000000000000000000
1057          cur tree    : 0000000000000000000000000000000000000000
1058          commit clock: 0
1059          last pack   :
1060
1061
1062        -------------------
1063        END OF CRASH REPORT
1064====
1065
1066Tips and Tricks
1067---------------
1068The following tips and tricks have been collected from various
1069users of fast-import, and are offered here as suggestions.
1070
1071Use One Mark Per Commit
1072~~~~~~~~~~~~~~~~~~~~~~~
1073When doing a repository conversion, use a unique mark per commit
1074(`mark :<n>`) and supply the \--export-marks option on the command
1075line.  fast-import will dump a file which lists every mark and the Git
1076object SHA-1 that corresponds to it.  If the frontend can tie
1077the marks back to the source repository, it is easy to verify the
1078accuracy and completeness of the import by comparing each Git
1079commit to the corresponding source revision.
1080
1081Coming from a system such as Perforce or Subversion this should be
1082quite simple, as the fast-import mark can also be the Perforce changeset
1083number or the Subversion revision number.
1084
1085Freely Skip Around Branches
1086~~~~~~~~~~~~~~~~~~~~~~~~~~~
1087Don't bother trying to optimize the frontend to stick to one branch
1088at a time during an import.  Although doing so might be slightly
1089faster for fast-import, it tends to increase the complexity of the frontend
1090code considerably.
1091
1092The branch LRU builtin to fast-import tends to behave very well, and the
1093cost of activating an inactive branch is so low that bouncing around
1094between branches has virtually no impact on import performance.
1095
1096Handling Renames
1097~~~~~~~~~~~~~~~~
1098When importing a renamed file or directory, simply delete the old
1099name(s) and modify the new name(s) during the corresponding commit.
1100Git performs rename detection after-the-fact, rather than explicitly
1101during a commit.
1102
1103Use Tag Fixup Branches
1104~~~~~~~~~~~~~~~~~~~~~~
1105Some other SCM systems let the user create a tag from multiple
1106files which are not from the same commit/changeset.  Or to create
1107tags which are a subset of the files available in the repository.
1108
1109Importing these tags as-is in Git is impossible without making at
1110least one commit which ``fixes up'' the files to match the content
1111of the tag.  Use fast-import's `reset` command to reset a dummy branch
1112outside of your normal branch space to the base commit for the tag,
1113then commit one or more file fixup commits, and finally tag the
1114dummy branch.
1115
1116For example since all normal branches are stored under `refs/heads/`
1117name the tag fixup branch `TAG_FIXUP`.  This way it is impossible for
1118the fixup branch used by the importer to have namespace conflicts
1119with real branches imported from the source (the name `TAG_FIXUP`
1120is not `refs/heads/TAG_FIXUP`).
1121
1122When committing fixups, consider using `merge` to connect the
1123commit(s) which are supplying file revisions to the fixup branch.
1124Doing so will allow tools such as 'git blame' to track
1125through the real commit history and properly annotate the source
1126files.
1127
1128After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP`
1129to remove the dummy branch.
1130
1131Import Now, Repack Later
1132~~~~~~~~~~~~~~~~~~~~~~~~
1133As soon as fast-import completes the Git repository is completely valid
1134and ready for use.  Typically this takes only a very short time,
1135even for considerably large projects (100,000+ commits).
1136
1137However repacking the repository is necessary to improve data
1138locality and access performance.  It can also take hours on extremely
1139large projects (especially if -f and a large \--window parameter is
1140used).  Since repacking is safe to run alongside readers and writers,
1141run the repack in the background and let it finish when it finishes.
1142There is no reason to wait to explore your new Git project!
1143
1144If you choose to wait for the repack, don't try to run benchmarks
1145or performance tests until repacking is completed.  fast-import outputs
1146suboptimal packfiles that are simply never seen in real use
1147situations.
1148
1149Repacking Historical Data
1150~~~~~~~~~~~~~~~~~~~~~~~~~
1151If you are repacking very old imported data (e.g. older than the
1152last year), consider expending some extra CPU time and supplying
1153\--window=50 (or higher) when you run 'git repack'.
1154This will take longer, but will also produce a smaller packfile.
1155You only need to expend the effort once, and everyone using your
1156project will benefit from the smaller repository.
1157
1158Include Some Progress Messages
1159~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1160Every once in a while have your frontend emit a `progress` message
1161to fast-import.  The contents of the messages are entirely free-form,
1162so one suggestion would be to output the current month and year
1163each time the current commit date moves into the next month.
1164Your users will feel better knowing how much of the data stream
1165has been processed.
1166
1167
1168Packfile Optimization
1169---------------------
1170When packing a blob fast-import always attempts to deltify against the last
1171blob written.  Unless specifically arranged for by the frontend,
1172this will probably not be a prior version of the same file, so the
1173generated delta will not be the smallest possible.  The resulting
1174packfile will be compressed, but will not be optimal.
1175
1176Frontends which have efficient access to all revisions of a
1177single file (for example reading an RCS/CVS ,v file) can choose
1178to supply all revisions of that file as a sequence of consecutive
1179`blob` commands.  This allows fast-import to deltify the different file
1180revisions against each other, saving space in the final packfile.
1181Marks can be used to later identify individual file revisions during
1182a sequence of `commit` commands.
1183
1184The packfile(s) created by fast-import do not encourage good disk access
1185patterns.  This is caused by fast-import writing the data in the order
1186it is received on standard input, while Git typically organizes
1187data within packfiles to make the most recent (current tip) data
1188appear before historical data.  Git also clusters commits together,
1189speeding up revision traversal through better cache locality.
1190
1191For this reason it is strongly recommended that users repack the
1192repository with `git repack -a -d` after fast-import completes, allowing
1193Git to reorganize the packfiles for faster data access.  If blob
1194deltas are suboptimal (see above) then also adding the `-f` option
1195to force recomputation of all deltas can significantly reduce the
1196final packfile size (30-50% smaller can be quite typical).
1197
1198
1199Memory Utilization
1200------------------
1201There are a number of factors which affect how much memory fast-import
1202requires to perform an import.  Like critical sections of core
1203Git, fast-import uses its own memory allocators to amortize any overheads
1204associated with malloc.  In practice fast-import tends to amortize any
1205malloc overheads to 0, due to its use of large block allocations.
1206
1207per object
1208~~~~~~~~~~
1209fast-import maintains an in-memory structure for every object written in
1210this execution.  On a 32 bit system the structure is 32 bytes,
1211on a 64 bit system the structure is 40 bytes (due to the larger
1212pointer sizes).  Objects in the table are not deallocated until
1213fast-import terminates.  Importing 2 million objects on a 32 bit system
1214will require approximately 64 MiB of memory.
1215
1216The object table is actually a hashtable keyed on the object name
1217(the unique SHA-1).  This storage configuration allows fast-import to reuse
1218an existing or already written object and avoid writing duplicates
1219to the output packfile.  Duplicate blobs are surprisingly common
1220in an import, typically due to branch merges in the source.
1221
1222per mark
1223~~~~~~~~
1224Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
1225bytes, depending on pointer size) per mark.  Although the array
1226is sparse, frontends are still strongly encouraged to use marks
1227between 1 and n, where n is the total number of marks required for
1228this import.
1229
1230per branch
1231~~~~~~~~~~
1232Branches are classified as active and inactive.  The memory usage
1233of the two classes is significantly different.
1234
1235Inactive branches are stored in a structure which uses 96 or 120
1236bytes (32 bit or 64 bit systems, respectively), plus the length of
1237the branch name (typically under 200 bytes), per branch.  fast-import will
1238easily handle as many as 10,000 inactive branches in under 2 MiB
1239of memory.
1240
1241Active branches have the same overhead as inactive branches, but
1242also contain copies of every tree that has been recently modified on
1243that branch.  If subtree `include` has not been modified since the
1244branch became active, its contents will not be loaded into memory,
1245but if subtree `src` has been modified by a commit since the branch
1246became active, then its contents will be loaded in memory.
1247
1248As active branches store metadata about the files contained on that
1249branch, their in-memory storage size can grow to a considerable size
1250(see below).
1251
1252fast-import automatically moves active branches to inactive status based on
1253a simple least-recently-used algorithm.  The LRU chain is updated on
1254each `commit` command.  The maximum number of active branches can be
1255increased or decreased on the command line with \--active-branches=.
1256
1257per active tree
1258~~~~~~~~~~~~~~~
1259Trees (aka directories) use just 12 bytes of memory on top of the
1260memory required for their entries (see ``per active file'' below).
1261The cost of a tree is virtually 0, as its overhead amortizes out
1262over the individual file entries.
1263
1264per active file entry
1265~~~~~~~~~~~~~~~~~~~~~
1266Files (and pointers to subtrees) within active trees require 52 or 64
1267bytes (32/64 bit platforms) per entry.  To conserve space, file and
1268tree names are pooled in a common string table, allowing the filename
1269``Makefile'' to use just 16 bytes (after including the string header
1270overhead) no matter how many times it occurs within the project.
1271
1272The active branch LRU, when coupled with the filename string pool
1273and lazy loading of subtrees, allows fast-import to efficiently import
1274projects with 2,000+ branches and 45,114+ files in a very limited
1275memory footprint (less than 2.7 MiB per active branch).
1276
1277Signals
1278-------
1279Sending *SIGUSR1* to the 'git fast-import' process ends the current
1280packfile early, simulating a `checkpoint` command.  The impatient
1281operator can use this facility to peek at the objects and refs from an
1282import in progress, at the cost of some added running time and worse
1283compression.
1284
1285Author
1286------
1287Written by Shawn O. Pearce <spearce@spearce.org>.
1288
1289Documentation
1290--------------
1291Documentation by Shawn O. Pearce <spearce@spearce.org>.
1292
1293GIT
1294---
1295Part of the linkgit:git[1] suite