Documentation / git-fast-import.txton commit Merge branch 'mm/maint-merge-ff-error-message-fix' (045c050)
   1git-fast-import(1)
   2==================
   3
   4NAME
   5----
   6git-fast-import - Backend for fast Git data importers
   7
   8
   9SYNOPSIS
  10--------
  11frontend | 'git fast-import' [options]
  12
  13DESCRIPTION
  14-----------
  15This program is usually not what the end user wants to run directly.
  16Most end users want to use one of the existing frontend programs,
  17which parses a specific type of foreign source and feeds the contents
  18stored there to 'git-fast-import'.
  19
  20fast-import reads a mixed command/data stream from standard input and
  21writes one or more packfiles directly into the current repository.
  22When EOF is received on standard input, fast import writes out
  23updated branch and tag refs, fully updating the current repository
  24with the newly imported data.
  25
  26The fast-import backend itself can import into an empty repository (one that
  27has already been initialized by 'git-init') or incrementally
  28update an existing populated repository.  Whether or not incremental
  29imports are supported from a particular foreign source depends on
  30the frontend program in use.
  31
  32
  33OPTIONS
  34-------
  35--date-format=<fmt>::
  36        Specify the type of dates the frontend will supply to
  37        fast-import within `author`, `committer` and `tagger` commands.
  38        See ``Date Formats'' below for details about which formats
  39        are supported, and their syntax.
  40
  41--force::
  42        Force updating modified existing branches, even if doing
  43        so would cause commits to be lost (as the new commit does
  44        not contain the old commit).
  45
  46--max-pack-size=<n>::
  47        Maximum size of each output packfile, expressed in MiB.
  48        The default is 4096 (4 GiB) as that is the maximum allowed
  49        packfile size (due to file format limitations). Some
  50        importers may wish to lower this, such as to ensure the
  51        resulting packfiles fit on CDs.
  52
  53--depth=<n>::
  54        Maximum delta depth, for blob and tree deltification.
  55        Default is 10.
  56
  57--active-branches=<n>::
  58        Maximum number of branches to maintain active at once.
  59        See ``Memory Utilization'' below for details.  Default is 5.
  60
  61--export-marks=<file>::
  62        Dumps the internal marks table to <file> when complete.
  63        Marks are written one per line as `:markid SHA-1`.
  64        Frontends can use this file to validate imports after they
  65        have been completed, or to save the marks table across
  66        incremental runs.  As <file> is only opened and truncated
  67        at checkpoint (or completion) the same path can also be
  68        safely given to \--import-marks.
  69
  70--import-marks=<file>::
  71        Before processing any input, load the marks specified in
  72        <file>.  The input file must exist, must be readable, and
  73        must use the same format as produced by \--export-marks.
  74        Multiple options may be supplied to import more than one
  75        set of marks.  If a mark is defined to different values,
  76        the last file wins.
  77
  78--export-pack-edges=<file>::
  79        After creating a packfile, print a line of data to
  80        <file> listing the filename of the packfile and the last
  81        commit on each branch that was written to that packfile.
  82        This information may be useful after importing projects
  83        whose total object set exceeds the 4 GiB packfile limit,
  84        as these commits can be used as edge points during calls
  85        to 'git-pack-objects'.
  86
  87--quiet::
  88        Disable all non-fatal output, making fast-import silent when it
  89        is successful.  This option disables the output shown by
  90        \--stats.
  91
  92--stats::
  93        Display some basic statistics about the objects fast-import has
  94        created, the packfiles they were stored into, and the
  95        memory used by fast-import during this run.  Showing this output
  96        is currently the default, but can be disabled with \--quiet.
  97
  98
  99Performance
 100-----------
 101The design of fast-import allows it to import large projects in a minimum
 102amount of memory usage and processing time.  Assuming the frontend
 103is able to keep up with fast-import and feed it a constant stream of data,
 104import times for projects holding 10+ years of history and containing
 105100,000+ individual commits are generally completed in just 1-2
 106hours on quite modest (~$2,000 USD) hardware.
 107
 108Most bottlenecks appear to be in foreign source data access (the
 109source just cannot extract revisions fast enough) or disk IO (fast-import
 110writes as fast as the disk will take the data).  Imports will run
 111faster if the source data is stored on a different drive than the
 112destination Git repository (due to less IO contention).
 113
 114
 115Development Cost
 116----------------
 117A typical frontend for fast-import tends to weigh in at approximately 200
 118lines of Perl/Python/Ruby code.  Most developers have been able to
 119create working importers in just a couple of hours, even though it
 120is their first exposure to fast-import, and sometimes even to Git.  This is
 121an ideal situation, given that most conversion tools are throw-away
 122(use once, and never look back).
 123
 124
 125Parallel Operation
 126------------------
 127Like 'git-push' or 'git-fetch', imports handled by fast-import are safe to
 128run alongside parallel `git repack -a -d` or `git gc` invocations,
 129or any other Git operation (including 'git-prune', as loose objects
 130are never used by fast-import).
 131
 132fast-import does not lock the branch or tag refs it is actively importing.
 133After the import, during its ref update phase, fast-import tests each
 134existing branch ref to verify the update will be a fast-forward
 135update (the commit stored in the ref is contained in the new
 136history of the commit to be written).  If the update is not a
 137fast-forward update, fast-import will skip updating that ref and instead
 138prints a warning message.  fast-import will always attempt to update all
 139branch refs, and does not stop on the first failure.
 140
 141Branch updates can be forced with \--force, but its recommended that
 142this only be used on an otherwise quiet repository.  Using \--force
 143is not necessary for an initial import into an empty repository.
 144
 145
 146Technical Discussion
 147--------------------
 148fast-import tracks a set of branches in memory.  Any branch can be created
 149or modified at any point during the import process by sending a
 150`commit` command on the input stream.  This design allows a frontend
 151program to process an unlimited number of branches simultaneously,
 152generating commits in the order they are available from the source
 153data.  It also simplifies the frontend programs considerably.
 154
 155fast-import does not use or alter the current working directory, or any
 156file within it.  (It does however update the current Git repository,
 157as referenced by `GIT_DIR`.)  Therefore an import frontend may use
 158the working directory for its own purposes, such as extracting file
 159revisions from the foreign source.  This ignorance of the working
 160directory also allows fast-import to run very quickly, as it does not
 161need to perform any costly file update operations when switching
 162between branches.
 163
 164Input Format
 165------------
 166With the exception of raw file data (which Git does not interpret)
 167the fast-import input format is text (ASCII) based.  This text based
 168format simplifies development and debugging of frontend programs,
 169especially when a higher level language such as Perl, Python or
 170Ruby is being used.
 171
 172fast-import is very strict about its input.  Where we say SP below we mean
 173*exactly* one space.  Likewise LF means one (and only one) linefeed.
 174Supplying additional whitespace characters will cause unexpected
 175results, such as branch names or file names with leading or trailing
 176spaces in their name, or early termination of fast-import when it encounters
 177unexpected input.
 178
 179Stream Comments
 180~~~~~~~~~~~~~~~
 181To aid in debugging frontends fast-import ignores any line that
 182begins with `#` (ASCII pound/hash) up to and including the line
 183ending `LF`.  A comment line may contain any sequence of bytes
 184that does not contain an LF and therefore may be used to include
 185any detailed debugging information that might be specific to the
 186frontend and useful when inspecting a fast-import data stream.
 187
 188Date Formats
 189~~~~~~~~~~~~
 190The following date formats are supported.  A frontend should select
 191the format it will use for this import by passing the format name
 192in the \--date-format=<fmt> command line option.
 193
 194`raw`::
 195        This is the Git native format and is `<time> SP <offutc>`.
 196        It is also fast-import's default format, if \--date-format was
 197        not specified.
 198+
 199The time of the event is specified by `<time>` as the number of
 200seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
 201written as an ASCII decimal integer.
 202+
 203The local offset is specified by `<offutc>` as a positive or negative
 204offset from UTC.  For example EST (which is 5 hours behind UTC)
 205would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''.
 206The local offset does not affect `<time>`; it is used only as an
 207advisement to help formatting routines display the timestamp.
 208+
 209If the local offset is not available in the source material, use
 210``+0000'', or the most common local offset.  For example many
 211organizations have a CVS repository which has only ever been accessed
 212by users who are located in the same location and timezone.  In this
 213case a reasonable offset from UTC could be assumed.
 214+
 215Unlike the `rfc2822` format, this format is very strict.  Any
 216variation in formatting will cause fast-import to reject the value.
 217
 218`rfc2822`::
 219        This is the standard email format as described by RFC 2822.
 220+
 221An example value is ``Tue Feb 6 11:22:18 2007 -0500''.  The Git
 222parser is accurate, but a little on the lenient side.  It is the
 223same parser used by 'git-am' when applying patches
 224received from email.
 225+
 226Some malformed strings may be accepted as valid dates.  In some of
 227these cases Git will still be able to obtain the correct date from
 228the malformed string.  There are also some types of malformed
 229strings which Git will parse wrong, and yet consider valid.
 230Seriously malformed strings will be rejected.
 231+
 232Unlike the `raw` format above, the timezone/UTC offset information
 233contained in an RFC 2822 date string is used to adjust the date
 234value to UTC prior to storage.  Therefore it is important that
 235this information be as accurate as possible.
 236+
 237If the source material uses RFC 2822 style dates,
 238the frontend should let fast-import handle the parsing and conversion
 239(rather than attempting to do it itself) as the Git parser has
 240been well tested in the wild.
 241+
 242Frontends should prefer the `raw` format if the source material
 243already uses UNIX-epoch format, can be coaxed to give dates in that
 244format, or its format is easily convertible to it, as there is no
 245ambiguity in parsing.
 246
 247`now`::
 248        Always use the current time and timezone.  The literal
 249        `now` must always be supplied for `<when>`.
 250+
 251This is a toy format.  The current time and timezone of this system
 252is always copied into the identity string at the time it is being
 253created by fast-import.  There is no way to specify a different time or
 254timezone.
 255+
 256This particular format is supplied as its short to implement and
 257may be useful to a process that wants to create a new commit
 258right now, without needing to use a working directory or
 259'git-update-index'.
 260+
 261If separate `author` and `committer` commands are used in a `commit`
 262the timestamps may not match, as the system clock will be polled
 263twice (once for each command).  The only way to ensure that both
 264author and committer identity information has the same timestamp
 265is to omit `author` (thus copying from `committer`) or to use a
 266date format other than `now`.
 267
 268Commands
 269~~~~~~~~
 270fast-import accepts several commands to update the current repository
 271and control the current import process.  More detailed discussion
 272(with examples) of each command follows later.
 273
 274`commit`::
 275        Creates a new branch or updates an existing branch by
 276        creating a new commit and updating the branch to point at
 277        the newly created commit.
 278
 279`tag`::
 280        Creates an annotated tag object from an existing commit or
 281        branch.  Lightweight tags are not supported by this command,
 282        as they are not recommended for recording meaningful points
 283        in time.
 284
 285`reset`::
 286        Reset an existing branch (or a new branch) to a specific
 287        revision.  This command must be used to change a branch to
 288        a specific revision without making a commit on it.
 289
 290`blob`::
 291        Convert raw file data into a blob, for future use in a
 292        `commit` command.  This command is optional and is not
 293        needed to perform an import.
 294
 295`checkpoint`::
 296        Forces fast-import to close the current packfile, generate its
 297        unique SHA-1 checksum and index, and start a new packfile.
 298        This command is optional and is not needed to perform
 299        an import.
 300
 301`progress`::
 302        Causes fast-import to echo the entire line to its own
 303        standard output.  This command is optional and is not needed
 304        to perform an import.
 305
 306`commit`
 307~~~~~~~~
 308Create or update a branch with a new commit, recording one logical
 309change to the project.
 310
 311....
 312        'commit' SP <ref> LF
 313        mark?
 314        ('author' SP <name> SP LT <email> GT SP <when> LF)?
 315        'committer' SP <name> SP LT <email> GT SP <when> LF
 316        data
 317        ('from' SP <committish> LF)?
 318        ('merge' SP <committish> LF)?
 319        (filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)*
 320        LF?
 321....
 322
 323where `<ref>` is the name of the branch to make the commit on.
 324Typically branch names are prefixed with `refs/heads/` in
 325Git, so importing the CVS branch symbol `RELENG-1_0` would use
 326`refs/heads/RELENG-1_0` for the value of `<ref>`.  The value of
 327`<ref>` must be a valid refname in Git.  As `LF` is not valid in
 328a Git refname, no quoting or escaping syntax is supported here.
 329
 330A `mark` command may optionally appear, requesting fast-import to save a
 331reference to the newly created commit for future use by the frontend
 332(see below for format).  It is very common for frontends to mark
 333every commit they create, thereby allowing future branch creation
 334from any imported commit.
 335
 336The `data` command following `committer` must supply the commit
 337message (see below for `data` command syntax).  To import an empty
 338commit message use a 0 length data.  Commit messages are free-form
 339and are not interpreted by Git.  Currently they must be encoded in
 340UTF-8, as fast-import does not permit other encodings to be specified.
 341
 342Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`,
 343`filedeleteall` and `notemodify` commands
 344may be included to update the contents of the branch prior to
 345creating the commit.  These commands may be supplied in any order.
 346However it is recommended that a `filedeleteall` command precede
 347all `filemodify`, `filecopy`, `filerename` and `notemodify` commands in
 348the same commit, as `filedeleteall` wipes the branch clean (see below).
 349
 350The `LF` after the command is optional (it used to be required).
 351
 352`author`
 353^^^^^^^^
 354An `author` command may optionally appear, if the author information
 355might differ from the committer information.  If `author` is omitted
 356then fast-import will automatically use the committer's information for
 357the author portion of the commit.  See below for a description of
 358the fields in `author`, as they are identical to `committer`.
 359
 360`committer`
 361^^^^^^^^^^^
 362The `committer` command indicates who made this commit, and when
 363they made it.
 364
 365Here `<name>` is the person's display name (for example
 366``Com M Itter'') and `<email>` is the person's email address
 367(``cm@example.com'').  `LT` and `GT` are the literal less-than (\x3c)
 368and greater-than (\x3e) symbols.  These are required to delimit
 369the email address from the other fields in the line.  Note that
 370`<name>` is free-form and may contain any sequence of bytes, except
 371`LT` and `LF`.  It is typically UTF-8 encoded.
 372
 373The time of the change is specified by `<when>` using the date format
 374that was selected by the \--date-format=<fmt> command line option.
 375See ``Date Formats'' above for the set of supported formats, and
 376their syntax.
 377
 378`from`
 379^^^^^^
 380The `from` command is used to specify the commit to initialize
 381this branch from.  This revision will be the first ancestor of the
 382new commit.
 383
 384Omitting the `from` command in the first commit of a new branch
 385will cause fast-import to create that commit with no ancestor. This
 386tends to be desired only for the initial commit of a project.
 387If the frontend creates all files from scratch when making a new
 388branch, a `merge` command may be used instead of `from` to start
 389the commit with an empty tree.
 390Omitting the `from` command on existing branches is usually desired,
 391as the current commit on that branch is automatically assumed to
 392be the first ancestor of the new commit.
 393
 394As `LF` is not valid in a Git refname or SHA-1 expression, no
 395quoting or escaping syntax is supported within `<committish>`.
 396
 397Here `<committish>` is any of the following:
 398
 399* The name of an existing branch already in fast-import's internal branch
 400  table.  If fast-import doesn't know the name, its treated as a SHA-1
 401  expression.
 402
 403* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
 404+
 405The reason fast-import uses `:` to denote a mark reference is this character
 406is not legal in a Git branch name.  The leading `:` makes it easy
 407to distinguish between the mark 42 (`:42`) and the branch 42 (`42`
 408or `refs/heads/42`), or an abbreviated SHA-1 which happened to
 409consist only of base-10 digits.
 410+
 411Marks must be declared (via `mark`) before they can be used.
 412
 413* A complete 40 byte or abbreviated commit SHA-1 in hex.
 414
 415* Any valid Git SHA-1 expression that resolves to a commit.  See
 416  ``SPECIFYING REVISIONS'' in linkgit:git-rev-parse[1] for details.
 417
 418The special case of restarting an incremental import from the
 419current branch value should be written as:
 420----
 421        from refs/heads/branch^0
 422----
 423The `{caret}0` suffix is necessary as fast-import does not permit a branch to
 424start from itself, and the branch is created in memory before the
 425`from` command is even read from the input.  Adding `{caret}0` will force
 426fast-import to resolve the commit through Git's revision parsing library,
 427rather than its internal branch table, thereby loading in the
 428existing value of the branch.
 429
 430`merge`
 431^^^^^^^
 432Includes one additional ancestor commit.  If the `from` command is
 433omitted when creating a new branch, the first `merge` commit will be
 434the first ancestor of the current commit, and the branch will start
 435out with no files.  An unlimited number of `merge` commands per
 436commit are permitted by fast-import, thereby establishing an n-way merge.
 437However Git's other tools never create commits with more than 15
 438additional ancestors (forming a 16-way merge).  For this reason
 439it is suggested that frontends do not use more than 15 `merge`
 440commands per commit; 16, if starting a new, empty branch.
 441
 442Here `<committish>` is any of the commit specification expressions
 443also accepted by `from` (see above).
 444
 445`filemodify`
 446^^^^^^^^^^^^
 447Included in a `commit` command to add a new file or change the
 448content of an existing file.  This command has two different means
 449of specifying the content of the file.
 450
 451External data format::
 452        The data content for the file was already supplied by a prior
 453        `blob` command.  The frontend just needs to connect it.
 454+
 455....
 456        'M' SP <mode> SP <dataref> SP <path> LF
 457....
 458+
 459Here `<dataref>` can be either a mark reference (`:<idnum>`)
 460set by a prior `blob` command, or a full 40-byte SHA-1 of an
 461existing Git blob object.
 462
 463Inline data format::
 464        The data content for the file has not been supplied yet.
 465        The frontend wants to supply it as part of this modify
 466        command.
 467+
 468....
 469        'M' SP <mode> SP 'inline' SP <path> LF
 470        data
 471....
 472+
 473See below for a detailed description of the `data` command.
 474
 475In both formats `<mode>` is the type of file entry, specified
 476in octal.  Git only supports the following modes:
 477
 478* `100644` or `644`: A normal (not-executable) file.  The majority
 479  of files in most projects use this mode.  If in doubt, this is
 480  what you want.
 481* `100755` or `755`: A normal, but executable, file.
 482* `120000`: A symlink, the content of the file will be the link target.
 483* `160000`: A gitlink, SHA-1 of the object refers to a commit in
 484  another repository. Git links can only be specified by SHA or through
 485  a commit mark. They are used to implement submodules.
 486
 487In both formats `<path>` is the complete path of the file to be added
 488(if not already existing) or modified (if already existing).
 489
 490A `<path>` string must use UNIX-style directory separators (forward
 491slash `/`), may contain any byte other than `LF`, and must not
 492start with double quote (`"`).
 493
 494If an `LF` or double quote must be encoded into `<path>` shell-style
 495quoting should be used, e.g. `"path/with\n and \" in it"`.
 496
 497The value of `<path>` must be in canonical form. That is it must not:
 498
 499* contain an empty directory component (e.g. `foo//bar` is invalid),
 500* end with a directory separator (e.g. `foo/` is invalid),
 501* start with a directory separator (e.g. `/foo` is invalid),
 502* contain the special component `.` or `..` (e.g. `foo/./bar` and
 503  `foo/../bar` are invalid).
 504
 505It is recommended that `<path>` always be encoded using UTF-8.
 506
 507`filedelete`
 508^^^^^^^^^^^^
 509Included in a `commit` command to remove a file or recursively
 510delete an entire directory from the branch.  If the file or directory
 511removal makes its parent directory empty, the parent directory will
 512be automatically removed too.  This cascades up the tree until the
 513first non-empty directory or the root is reached.
 514
 515....
 516        'D' SP <path> LF
 517....
 518
 519here `<path>` is the complete path of the file or subdirectory to
 520be removed from the branch.
 521See `filemodify` above for a detailed description of `<path>`.
 522
 523`filecopy`
 524^^^^^^^^^^^^
 525Recursively copies an existing file or subdirectory to a different
 526location within the branch.  The existing file or directory must
 527exist.  If the destination exists it will be completely replaced
 528by the content copied from the source.
 529
 530....
 531        'C' SP <path> SP <path> LF
 532....
 533
 534here the first `<path>` is the source location and the second
 535`<path>` is the destination.  See `filemodify` above for a detailed
 536description of what `<path>` may look like.  To use a source path
 537that contains SP the path must be quoted.
 538
 539A `filecopy` command takes effect immediately.  Once the source
 540location has been copied to the destination any future commands
 541applied to the source location will not impact the destination of
 542the copy.
 543
 544`filerename`
 545^^^^^^^^^^^^
 546Renames an existing file or subdirectory to a different location
 547within the branch.  The existing file or directory must exist. If
 548the destination exists it will be replaced by the source directory.
 549
 550....
 551        'R' SP <path> SP <path> LF
 552....
 553
 554here the first `<path>` is the source location and the second
 555`<path>` is the destination.  See `filemodify` above for a detailed
 556description of what `<path>` may look like.  To use a source path
 557that contains SP the path must be quoted.
 558
 559A `filerename` command takes effect immediately.  Once the source
 560location has been renamed to the destination any future commands
 561applied to the source location will create new files there and not
 562impact the destination of the rename.
 563
 564Note that a `filerename` is the same as a `filecopy` followed by a
 565`filedelete` of the source location.  There is a slight performance
 566advantage to using `filerename`, but the advantage is so small
 567that it is never worth trying to convert a delete/add pair in
 568source material into a rename for fast-import.  This `filerename`
 569command is provided just to simplify frontends that already have
 570rename information and don't want bother with decomposing it into a
 571`filecopy` followed by a `filedelete`.
 572
 573`filedeleteall`
 574^^^^^^^^^^^^^^^
 575Included in a `commit` command to remove all files (and also all
 576directories) from the branch.  This command resets the internal
 577branch structure to have no files in it, allowing the frontend
 578to subsequently add all interesting files from scratch.
 579
 580....
 581        'deleteall' LF
 582....
 583
 584This command is extremely useful if the frontend does not know
 585(or does not care to know) what files are currently on the branch,
 586and therefore cannot generate the proper `filedelete` commands to
 587update the content.
 588
 589Issuing a `filedeleteall` followed by the needed `filemodify`
 590commands to set the correct content will produce the same results
 591as sending only the needed `filemodify` and `filedelete` commands.
 592The `filedeleteall` approach may however require fast-import to use slightly
 593more memory per active branch (less than 1 MiB for even most large
 594projects); so frontends that can easily obtain only the affected
 595paths for a commit are encouraged to do so.
 596
 597`notemodify`
 598^^^^^^^^^^^^
 599Included in a `commit` command to add a new note (annotating a given
 600commit) or change the content of an existing note.  This command has
 601two different means of specifying the content of the note.
 602
 603External data format::
 604        The data content for the note was already supplied by a prior
 605        `blob` command.  The frontend just needs to connect it to the
 606        commit that is to be annotated.
 607+
 608....
 609        'N' SP <dataref> SP <committish> LF
 610....
 611+
 612Here `<dataref>` can be either a mark reference (`:<idnum>`)
 613set by a prior `blob` command, or a full 40-byte SHA-1 of an
 614existing Git blob object.
 615
 616Inline data format::
 617        The data content for the note has not been supplied yet.
 618        The frontend wants to supply it as part of this modify
 619        command.
 620+
 621....
 622        'N' SP 'inline' SP <committish> LF
 623        data
 624....
 625+
 626See below for a detailed description of the `data` command.
 627
 628In both formats `<committish>` is any of the commit specification
 629expressions also accepted by `from` (see above).
 630
 631`mark`
 632~~~~~~
 633Arranges for fast-import to save a reference to the current object, allowing
 634the frontend to recall this object at a future point in time, without
 635knowing its SHA-1.  Here the current object is the object creation
 636command the `mark` command appears within.  This can be `commit`,
 637`tag`, and `blob`, but `commit` is the most common usage.
 638
 639....
 640        'mark' SP ':' <idnum> LF
 641....
 642
 643where `<idnum>` is the number assigned by the frontend to this mark.
 644The value of `<idnum>` is expressed as an ASCII decimal integer.
 645The value 0 is reserved and cannot be used as
 646a mark.  Only values greater than or equal to 1 may be used as marks.
 647
 648New marks are created automatically.  Existing marks can be moved
 649to another object simply by reusing the same `<idnum>` in another
 650`mark` command.
 651
 652`tag`
 653~~~~~
 654Creates an annotated tag referring to a specific commit.  To create
 655lightweight (non-annotated) tags see the `reset` command below.
 656
 657....
 658        'tag' SP <name> LF
 659        'from' SP <committish> LF
 660        'tagger' SP <name> SP LT <email> GT SP <when> LF
 661        data
 662....
 663
 664where `<name>` is the name of the tag to create.
 665
 666Tag names are automatically prefixed with `refs/tags/` when stored
 667in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
 668use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the
 669corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
 670
 671The value of `<name>` must be a valid refname in Git and therefore
 672may contain forward slashes.  As `LF` is not valid in a Git refname,
 673no quoting or escaping syntax is supported here.
 674
 675The `from` command is the same as in the `commit` command; see
 676above for details.
 677
 678The `tagger` command uses the same format as `committer` within
 679`commit`; again see above for details.
 680
 681The `data` command following `tagger` must supply the annotated tag
 682message (see below for `data` command syntax).  To import an empty
 683tag message use a 0 length data.  Tag messages are free-form and are
 684not interpreted by Git.  Currently they must be encoded in UTF-8,
 685as fast-import does not permit other encodings to be specified.
 686
 687Signing annotated tags during import from within fast-import is not
 688supported.  Trying to include your own PGP/GPG signature is not
 689recommended, as the frontend does not (easily) have access to the
 690complete set of bytes which normally goes into such a signature.
 691If signing is required, create lightweight tags from within fast-import with
 692`reset`, then create the annotated versions of those tags offline
 693with the standard 'git-tag' process.
 694
 695`reset`
 696~~~~~~~
 697Creates (or recreates) the named branch, optionally starting from
 698a specific revision.  The reset command allows a frontend to issue
 699a new `from` command for an existing branch, or to create a new
 700branch from an existing commit without creating a new commit.
 701
 702....
 703        'reset' SP <ref> LF
 704        ('from' SP <committish> LF)?
 705        LF?
 706....
 707
 708For a detailed description of `<ref>` and `<committish>` see above
 709under `commit` and `from`.
 710
 711The `LF` after the command is optional (it used to be required).
 712
 713The `reset` command can also be used to create lightweight
 714(non-annotated) tags.  For example:
 715
 716====
 717        reset refs/tags/938
 718        from :938
 719====
 720
 721would create the lightweight tag `refs/tags/938` referring to
 722whatever commit mark `:938` references.
 723
 724`blob`
 725~~~~~~
 726Requests writing one file revision to the packfile.  The revision
 727is not connected to any commit; this connection must be formed in
 728a subsequent `commit` command by referencing the blob through an
 729assigned mark.
 730
 731....
 732        'blob' LF
 733        mark?
 734        data
 735....
 736
 737The mark command is optional here as some frontends have chosen
 738to generate the Git SHA-1 for the blob on their own, and feed that
 739directly to `commit`.  This is typically more work than its worth
 740however, as marks are inexpensive to store and easy to use.
 741
 742`data`
 743~~~~~~
 744Supplies raw data (for use as blob/file content, commit messages, or
 745annotated tag messages) to fast-import.  Data can be supplied using an exact
 746byte count or delimited with a terminating line.  Real frontends
 747intended for production-quality conversions should always use the
 748exact byte count format, as it is more robust and performs better.
 749The delimited format is intended primarily for testing fast-import.
 750
 751Comment lines appearing within the `<raw>` part of `data` commands
 752are always taken to be part of the body of the data and are therefore
 753never ignored by fast-import.  This makes it safe to import any
 754file/message content whose lines might start with `#`.
 755
 756Exact byte count format::
 757        The frontend must specify the number of bytes of data.
 758+
 759....
 760        'data' SP <count> LF
 761        <raw> LF?
 762....
 763+
 764where `<count>` is the exact number of bytes appearing within
 765`<raw>`.  The value of `<count>` is expressed as an ASCII decimal
 766integer.  The `LF` on either side of `<raw>` is not
 767included in `<count>` and will not be included in the imported data.
 768+
 769The `LF` after `<raw>` is optional (it used to be required) but
 770recommended.  Always including it makes debugging a fast-import
 771stream easier as the next command always starts in column 0
 772of the next line, even if `<raw>` did not end with an `LF`.
 773
 774Delimited format::
 775        A delimiter string is used to mark the end of the data.
 776        fast-import will compute the length by searching for the delimiter.
 777        This format is primarily useful for testing and is not
 778        recommended for real data.
 779+
 780....
 781        'data' SP '<<' <delim> LF
 782        <raw> LF
 783        <delim> LF
 784        LF?
 785....
 786+
 787where `<delim>` is the chosen delimiter string.  The string `<delim>`
 788must not appear on a line by itself within `<raw>`, as otherwise
 789fast-import will think the data ends earlier than it really does.  The `LF`
 790immediately trailing `<raw>` is part of `<raw>`.  This is one of
 791the limitations of the delimited format, it is impossible to supply
 792a data chunk which does not have an LF as its last byte.
 793+
 794The `LF` after `<delim> LF` is optional (it used to be required).
 795
 796`checkpoint`
 797~~~~~~~~~~~~
 798Forces fast-import to close the current packfile, start a new one, and to
 799save out all current branch refs, tags and marks.
 800
 801....
 802        'checkpoint' LF
 803        LF?
 804....
 805
 806Note that fast-import automatically switches packfiles when the current
 807packfile reaches \--max-pack-size, or 4 GiB, whichever limit is
 808smaller.  During an automatic packfile switch fast-import does not update
 809the branch refs, tags or marks.
 810
 811As a `checkpoint` can require a significant amount of CPU time and
 812disk IO (to compute the overall pack SHA-1 checksum, generate the
 813corresponding index file, and update the refs) it can easily take
 814several minutes for a single `checkpoint` command to complete.
 815
 816Frontends may choose to issue checkpoints during extremely large
 817and long running imports, or when they need to allow another Git
 818process access to a branch.  However given that a 30 GiB Subversion
 819repository can be loaded into Git through fast-import in about 3 hours,
 820explicit checkpointing may not be necessary.
 821
 822The `LF` after the command is optional (it used to be required).
 823
 824`progress`
 825~~~~~~~~~~
 826Causes fast-import to print the entire `progress` line unmodified to
 827its standard output channel (file descriptor 1) when the command is
 828processed from the input stream.  The command otherwise has no impact
 829on the current import, or on any of fast-import's internal state.
 830
 831....
 832        'progress' SP <any> LF
 833        LF?
 834....
 835
 836The `<any>` part of the command may contain any sequence of bytes
 837that does not contain `LF`.  The `LF` after the command is optional.
 838Callers may wish to process the output through a tool such as sed to
 839remove the leading part of the line, for example:
 840
 841====
 842        frontend | git fast-import | sed 's/^progress //'
 843====
 844
 845Placing a `progress` command immediately after a `checkpoint` will
 846inform the reader when the `checkpoint` has been completed and it
 847can safely access the refs that fast-import updated.
 848
 849Crash Reports
 850-------------
 851If fast-import is supplied invalid input it will terminate with a
 852non-zero exit status and create a crash report in the top level of
 853the Git repository it was importing into.  Crash reports contain
 854a snapshot of the internal fast-import state as well as the most
 855recent commands that lead up to the crash.
 856
 857All recent commands (including stream comments, file changes and
 858progress commands) are shown in the command history within the crash
 859report, but raw file data and commit messages are excluded from the
 860crash report.  This exclusion saves space within the report file
 861and reduces the amount of buffering that fast-import must perform
 862during execution.
 863
 864After writing a crash report fast-import will close the current
 865packfile and export the marks table.  This allows the frontend
 866developer to inspect the repository state and resume the import from
 867the point where it crashed.  The modified branches and tags are not
 868updated during a crash, as the import did not complete successfully.
 869Branch and tag information can be found in the crash report and
 870must be applied manually if the update is needed.
 871
 872An example crash:
 873
 874====
 875        $ cat >in <<END_OF_INPUT
 876        # my very first test commit
 877        commit refs/heads/master
 878        committer Shawn O. Pearce <spearce> 19283 -0400
 879        # who is that guy anyway?
 880        data <<EOF
 881        this is my commit
 882        EOF
 883        M 644 inline .gitignore
 884        data <<EOF
 885        .gitignore
 886        EOF
 887        M 777 inline bob
 888        END_OF_INPUT
 889
 890        $ git fast-import <in
 891        fatal: Corrupt mode: M 777 inline bob
 892        fast-import: dumping crash report to .git/fast_import_crash_8434
 893
 894        $ cat .git/fast_import_crash_8434
 895        fast-import crash report:
 896            fast-import process: 8434
 897            parent process     : 1391
 898            at Sat Sep 1 00:58:12 2007
 899
 900        fatal: Corrupt mode: M 777 inline bob
 901
 902        Most Recent Commands Before Crash
 903        ---------------------------------
 904          # my very first test commit
 905          commit refs/heads/master
 906          committer Shawn O. Pearce <spearce> 19283 -0400
 907          # who is that guy anyway?
 908          data <<EOF
 909          M 644 inline .gitignore
 910          data <<EOF
 911        * M 777 inline bob
 912
 913        Active Branch LRU
 914        -----------------
 915            active_branches = 1 cur, 5 max
 916
 917          pos  clock name
 918          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 919           1)      0 refs/heads/master
 920
 921        Inactive Branches
 922        -----------------
 923        refs/heads/master:
 924          status      : active loaded dirty
 925          tip commit  : 0000000000000000000000000000000000000000
 926          old tree    : 0000000000000000000000000000000000000000
 927          cur tree    : 0000000000000000000000000000000000000000
 928          commit clock: 0
 929          last pack   :
 930
 931
 932        -------------------
 933        END OF CRASH REPORT
 934====
 935
 936Tips and Tricks
 937---------------
 938The following tips and tricks have been collected from various
 939users of fast-import, and are offered here as suggestions.
 940
 941Use One Mark Per Commit
 942~~~~~~~~~~~~~~~~~~~~~~~
 943When doing a repository conversion, use a unique mark per commit
 944(`mark :<n>`) and supply the \--export-marks option on the command
 945line.  fast-import will dump a file which lists every mark and the Git
 946object SHA-1 that corresponds to it.  If the frontend can tie
 947the marks back to the source repository, it is easy to verify the
 948accuracy and completeness of the import by comparing each Git
 949commit to the corresponding source revision.
 950
 951Coming from a system such as Perforce or Subversion this should be
 952quite simple, as the fast-import mark can also be the Perforce changeset
 953number or the Subversion revision number.
 954
 955Freely Skip Around Branches
 956~~~~~~~~~~~~~~~~~~~~~~~~~~~
 957Don't bother trying to optimize the frontend to stick to one branch
 958at a time during an import.  Although doing so might be slightly
 959faster for fast-import, it tends to increase the complexity of the frontend
 960code considerably.
 961
 962The branch LRU builtin to fast-import tends to behave very well, and the
 963cost of activating an inactive branch is so low that bouncing around
 964between branches has virtually no impact on import performance.
 965
 966Handling Renames
 967~~~~~~~~~~~~~~~~
 968When importing a renamed file or directory, simply delete the old
 969name(s) and modify the new name(s) during the corresponding commit.
 970Git performs rename detection after-the-fact, rather than explicitly
 971during a commit.
 972
 973Use Tag Fixup Branches
 974~~~~~~~~~~~~~~~~~~~~~~
 975Some other SCM systems let the user create a tag from multiple
 976files which are not from the same commit/changeset.  Or to create
 977tags which are a subset of the files available in the repository.
 978
 979Importing these tags as-is in Git is impossible without making at
 980least one commit which ``fixes up'' the files to match the content
 981of the tag.  Use fast-import's `reset` command to reset a dummy branch
 982outside of your normal branch space to the base commit for the tag,
 983then commit one or more file fixup commits, and finally tag the
 984dummy branch.
 985
 986For example since all normal branches are stored under `refs/heads/`
 987name the tag fixup branch `TAG_FIXUP`.  This way it is impossible for
 988the fixup branch used by the importer to have namespace conflicts
 989with real branches imported from the source (the name `TAG_FIXUP`
 990is not `refs/heads/TAG_FIXUP`).
 991
 992When committing fixups, consider using `merge` to connect the
 993commit(s) which are supplying file revisions to the fixup branch.
 994Doing so will allow tools such as 'git-blame' to track
 995through the real commit history and properly annotate the source
 996files.
 997
 998After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP`
 999to remove the dummy branch.
1000
1001Import Now, Repack Later
1002~~~~~~~~~~~~~~~~~~~~~~~~
1003As soon as fast-import completes the Git repository is completely valid
1004and ready for use.  Typically this takes only a very short time,
1005even for considerably large projects (100,000+ commits).
1006
1007However repacking the repository is necessary to improve data
1008locality and access performance.  It can also take hours on extremely
1009large projects (especially if -f and a large \--window parameter is
1010used).  Since repacking is safe to run alongside readers and writers,
1011run the repack in the background and let it finish when it finishes.
1012There is no reason to wait to explore your new Git project!
1013
1014If you choose to wait for the repack, don't try to run benchmarks
1015or performance tests until repacking is completed.  fast-import outputs
1016suboptimal packfiles that are simply never seen in real use
1017situations.
1018
1019Repacking Historical Data
1020~~~~~~~~~~~~~~~~~~~~~~~~~
1021If you are repacking very old imported data (e.g. older than the
1022last year), consider expending some extra CPU time and supplying
1023\--window=50 (or higher) when you run 'git-repack'.
1024This will take longer, but will also produce a smaller packfile.
1025You only need to expend the effort once, and everyone using your
1026project will benefit from the smaller repository.
1027
1028Include Some Progress Messages
1029~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1030Every once in a while have your frontend emit a `progress` message
1031to fast-import.  The contents of the messages are entirely free-form,
1032so one suggestion would be to output the current month and year
1033each time the current commit date moves into the next month.
1034Your users will feel better knowing how much of the data stream
1035has been processed.
1036
1037
1038Packfile Optimization
1039---------------------
1040When packing a blob fast-import always attempts to deltify against the last
1041blob written.  Unless specifically arranged for by the frontend,
1042this will probably not be a prior version of the same file, so the
1043generated delta will not be the smallest possible.  The resulting
1044packfile will be compressed, but will not be optimal.
1045
1046Frontends which have efficient access to all revisions of a
1047single file (for example reading an RCS/CVS ,v file) can choose
1048to supply all revisions of that file as a sequence of consecutive
1049`blob` commands.  This allows fast-import to deltify the different file
1050revisions against each other, saving space in the final packfile.
1051Marks can be used to later identify individual file revisions during
1052a sequence of `commit` commands.
1053
1054The packfile(s) created by fast-import do not encourage good disk access
1055patterns.  This is caused by fast-import writing the data in the order
1056it is received on standard input, while Git typically organizes
1057data within packfiles to make the most recent (current tip) data
1058appear before historical data.  Git also clusters commits together,
1059speeding up revision traversal through better cache locality.
1060
1061For this reason it is strongly recommended that users repack the
1062repository with `git repack -a -d` after fast-import completes, allowing
1063Git to reorganize the packfiles for faster data access.  If blob
1064deltas are suboptimal (see above) then also adding the `-f` option
1065to force recomputation of all deltas can significantly reduce the
1066final packfile size (30-50% smaller can be quite typical).
1067
1068
1069Memory Utilization
1070------------------
1071There are a number of factors which affect how much memory fast-import
1072requires to perform an import.  Like critical sections of core
1073Git, fast-import uses its own memory allocators to amortize any overheads
1074associated with malloc.  In practice fast-import tends to amortize any
1075malloc overheads to 0, due to its use of large block allocations.
1076
1077per object
1078~~~~~~~~~~
1079fast-import maintains an in-memory structure for every object written in
1080this execution.  On a 32 bit system the structure is 32 bytes,
1081on a 64 bit system the structure is 40 bytes (due to the larger
1082pointer sizes).  Objects in the table are not deallocated until
1083fast-import terminates.  Importing 2 million objects on a 32 bit system
1084will require approximately 64 MiB of memory.
1085
1086The object table is actually a hashtable keyed on the object name
1087(the unique SHA-1).  This storage configuration allows fast-import to reuse
1088an existing or already written object and avoid writing duplicates
1089to the output packfile.  Duplicate blobs are surprisingly common
1090in an import, typically due to branch merges in the source.
1091
1092per mark
1093~~~~~~~~
1094Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
1095bytes, depending on pointer size) per mark.  Although the array
1096is sparse, frontends are still strongly encouraged to use marks
1097between 1 and n, where n is the total number of marks required for
1098this import.
1099
1100per branch
1101~~~~~~~~~~
1102Branches are classified as active and inactive.  The memory usage
1103of the two classes is significantly different.
1104
1105Inactive branches are stored in a structure which uses 96 or 120
1106bytes (32 bit or 64 bit systems, respectively), plus the length of
1107the branch name (typically under 200 bytes), per branch.  fast-import will
1108easily handle as many as 10,000 inactive branches in under 2 MiB
1109of memory.
1110
1111Active branches have the same overhead as inactive branches, but
1112also contain copies of every tree that has been recently modified on
1113that branch.  If subtree `include` has not been modified since the
1114branch became active, its contents will not be loaded into memory,
1115but if subtree `src` has been modified by a commit since the branch
1116became active, then its contents will be loaded in memory.
1117
1118As active branches store metadata about the files contained on that
1119branch, their in-memory storage size can grow to a considerable size
1120(see below).
1121
1122fast-import automatically moves active branches to inactive status based on
1123a simple least-recently-used algorithm.  The LRU chain is updated on
1124each `commit` command.  The maximum number of active branches can be
1125increased or decreased on the command line with \--active-branches=.
1126
1127per active tree
1128~~~~~~~~~~~~~~~
1129Trees (aka directories) use just 12 bytes of memory on top of the
1130memory required for their entries (see ``per active file'' below).
1131The cost of a tree is virtually 0, as its overhead amortizes out
1132over the individual file entries.
1133
1134per active file entry
1135~~~~~~~~~~~~~~~~~~~~~
1136Files (and pointers to subtrees) within active trees require 52 or 64
1137bytes (32/64 bit platforms) per entry.  To conserve space, file and
1138tree names are pooled in a common string table, allowing the filename
1139``Makefile'' to use just 16 bytes (after including the string header
1140overhead) no matter how many times it occurs within the project.
1141
1142The active branch LRU, when coupled with the filename string pool
1143and lazy loading of subtrees, allows fast-import to efficiently import
1144projects with 2,000+ branches and 45,114+ files in a very limited
1145memory footprint (less than 2.7 MiB per active branch).
1146
1147
1148Author
1149------
1150Written by Shawn O. Pearce <spearce@spearce.org>.
1151
1152Documentation
1153--------------
1154Documentation by Shawn O. Pearce <spearce@spearce.org>.
1155
1156GIT
1157---
1158Part of the linkgit:git[1] suite