e204ea1b3ef60a2cc866b8eb54082dff08832f3d
   1git-fast-import(1)
   2==================
   3
   4NAME
   5----
   6git-fast-import - Backend for fast Git data importers.
   7
   8
   9SYNOPSIS
  10--------
  11frontend | 'git-fast-import' [options]
  12
  13DESCRIPTION
  14-----------
  15This program is usually not what the end user wants to run directly.
  16Most end users want to use one of the existing frontend programs,
  17which parses a specific type of foreign source and feeds the contents
  18stored there to git-fast-import (gfi).
  19
  20gfi reads a mixed command/data stream from standard input and
  21writes one or more packfiles directly into the current repository.
  22When EOF is received on standard input, fast import writes out
  23updated branch and tag refs, fully updating the current repository
  24with the newly imported data.
  25
  26The gfi backend itself can import into an empty repository (one that
  27has already been initialized by gitlink:git-init[1]) or incrementally
  28update an existing populated repository.  Whether or not incremental
  29imports are supported from a particular foreign source depends on
  30the frontend program in use.
  31
  32
  33OPTIONS
  34-------
  35--max-pack-size=<n>::
  36        Maximum size of each output packfile, expressed in MiB.
  37        The default is 4096 (4 GiB) as that is the maximum allowed
  38        packfile size (due to file format limitations). Some
  39        importers may wish to lower this, such as to ensure the
  40        resulting packfiles fit on CDs.
  41
  42--depth=<n>::
  43        Maximum delta depth, for blob and tree deltification.
  44        Default is 10.
  45
  46--active-branches=<n>::
  47        Maximum number of branches to maintain active at once.
  48        See ``Memory Utilization'' below for details.  Default is 5.
  49
  50--export-marks=<file>::
  51        Dumps the internal marks table to <file> when complete.
  52        Marks are written one per line as `:markid SHA-1`.
  53        Frontends can use this file to validate imports after they
  54        have been completed.
  55
  56
  57Performance
  58-----------
  59The design of gfi allows it to import large projects in a minimum
  60amount of memory usage and processing time.  Assuming the frontend
  61is able to keep up with gfi and feed it a constant stream of data,
  62import times for projects holding 10+ years of history and containing
  63100,000+ individual commits are generally completed in just 1-2
  64hours on quite modest (~$2,000 USD) hardware.
  65
  66Most bottlenecks appear to be in foreign source data access (the
  67source just cannot extract revisions fast enough) or disk IO (gfi
  68writes as fast as the disk will take the data).  Imports will run
  69faster if the source data is stored on a different drive than the
  70destination Git repository (due to less IO contention).
  71
  72
  73Development Cost
  74----------------
  75A typical frontend for gfi tends to weigh in at approximately 200
  76lines of Perl/Python/Ruby code.  Most developers have been able to
  77create working importers in just a couple of hours, even though it
  78is their first exposure to gfi, and sometimes even to Git.  This is
  79an ideal situation, given that most conversion tools are throw-away
  80(use once, and never look back).
  81
  82
  83Parallel Operation
  84------------------
  85Like `git-push` or `git-fetch`, imports handled by gfi are safe to
  86run alongside parallel `git repack -a -d` or `git gc` invocations,
  87or any other Git operation (including `git prune`, as loose objects
  88are never used by gfi).
  89
  90However, gfi does not lock the branch or tag refs it is actively
  91importing.  After EOF, during its ref update phase, gfi blindly
  92overwrites each imported branch or tag ref.  Consequently it is not
  93safe to modify refs that are currently being used by a running gfi
  94instance, as work could be lost when gfi overwrites the refs.
  95
  96
  97Technical Discussion
  98--------------------
  99gfi tracks a set of branches in memory.  Any branch can be created
 100or modified at any point during the import process by sending a
 101`commit` command on the input stream.  This design allows a frontend
 102program to process an unlimited number of branches simultaneously,
 103generating commits in the order they are available from the source
 104data.  It also simplifies the frontend programs considerably.
 105
 106gfi does not use or alter the current working directory, or any
 107file within it.  (It does however update the current Git repository,
 108as referenced by `GIT_DIR`.)  Therefore an import frontend may use
 109the working directory for its own purposes, such as extracting file
 110revisions from the foreign source.  This ignorance of the working
 111directory also allows gfi to run very quickly, as it does not
 112need to perform any costly file update operations when switching
 113between branches.
 114
 115Input Format
 116------------
 117With the exception of raw file data (which Git does not interpret)
 118the gfi input format is text (ASCII) based.  This text based
 119format simplifies development and debugging of frontend programs,
 120especially when a higher level language such as Perl, Python or
 121Ruby is being used.
 122
 123gfi is very strict about its input.  Where we say SP below we mean
 124*exactly* one space.  Likewise LF means one (and only one) linefeed.
 125Supplying additional whitespace characters will cause unexpected
 126results, such as branch names or file names with leading or trailing
 127spaces in their name, or early termination of gfi when it encounters
 128unexpected input.
 129
 130Commands
 131~~~~~~~~
 132gfi accepts several commands to update the current repository
 133and control the current import process.  More detailed discussion
 134(with examples) of each command follows later.
 135
 136`commit`::
 137        Creates a new branch or updates an existing branch by
 138        creating a new commit and updating the branch to point at
 139        the newly created commit.
 140
 141`tag`::
 142        Creates an annotated tag object from an existing commit or
 143        branch.  Lightweight tags are not supported by this command,
 144        as they are not recommended for recording meaningful points
 145        in time.
 146
 147`reset`::
 148        Reset an existing branch (or a new branch) to a specific
 149        revision.  This command must be used to change a branch to
 150        a specific revision without making a commit on it.
 151
 152`blob`::
 153        Convert raw file data into a blob, for future use in a
 154        `commit` command.  This command is optional and is not
 155        needed to perform an import.
 156
 157`checkpoint`::
 158        Forces gfi to close the current packfile, generate its
 159        unique SHA-1 checksum and index, and start a new packfile.
 160        This command is optional and is not needed to perform
 161        an import.
 162
 163`commit`
 164~~~~~~~~
 165Create or update a branch with a new commit, recording one logical
 166change to the project.
 167
 168....
 169        'commit' SP <ref> LF
 170        mark?
 171        ('author' SP <name> SP LT <email> GT SP <time> SP <tz> LF)?
 172        'committer' SP <name> SP LT <email> GT SP <time> SP <tz> LF
 173        data
 174        ('from' SP <committish> LF)?
 175        ('merge' SP <committish> LF)?
 176        (filemodify | filedelete)*
 177        LF
 178....
 179
 180where `<ref>` is the name of the branch to make the commit on.
 181Typically branch names are prefixed with `refs/heads/` in
 182Git, so importing the CVS branch symbol `RELENG-1_0` would use
 183`refs/heads/RELENG-1_0` for the value of `<ref>`.  The value of
 184`<ref>` must be a valid refname in Git.  As `LF` is not valid in
 185a Git refname, no quoting or escaping syntax is supported here.
 186
 187A `mark` command may optionally appear, requesting gfi to save a
 188reference to the newly created commit for future use by the frontend
 189(see below for format).  It is very common for frontends to mark
 190every commit they create, thereby allowing future branch creation
 191from any imported commit.
 192
 193The `data` command following `committer` must supply the commit
 194message (see below for `data` command syntax).  To import an empty
 195commit message use a 0 length data.  Commit messages are free-form
 196and are not interpreted by Git.  Currently they must be encoded in
 197UTF-8, as gfi does not permit other encodings to be specified.
 198
 199Zero or more `filemodify` and `filedelete` commands may be
 200included to update the contents of the branch prior to the commit.
 201These commands can be supplied in any order, gfi is not sensitive
 202to pathname or operation ordering.
 203
 204`author`
 205^^^^^^^^
 206An `author` command may optionally appear, if the author information
 207might differ from the committer information.  If `author` is omitted
 208then gfi will automatically use the committer's information for
 209the author portion of the commit.  See below for a description of
 210the fields in `author`, as they are identical to `committer`.
 211
 212`committer`
 213^^^^^^^^^^^
 214The `committer` command indicates who made this commit, and when
 215they made it.
 216
 217Here `<name>` is the person's display name (for example
 218``Com M Itter'') and `<email>` is the person's email address
 219(``cm@example.com'').  `LT` and `GT` are the literal less-than (\x3c)
 220and greater-than (\x3e) symbols.  These are required to delimit
 221the email address from the other fields in the line.  Note that
 222`<name>` is free-form and may contain any sequence of bytes, except
 223`LT` and `LF`.  It is typically UTF-8 encoded.
 224
 225The time of the change is specified by `<time>` as the number of
 226seconds since the UNIX epoc (midnight, Jan 1, 1970, UTC) and is
 227written in base-10 notation using US-ASCII digits.  The committer's
 228timezone is specified by `<tz>` as a positive or negative offset
 229from UTC.  For example EST (which is typically 5 hours behind GMT)
 230would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
 231
 232`from`
 233^^^^^^
 234Only valid for the first commit made on this branch by this
 235gfi process.  The `from` command is used to specify the commit
 236to initialize this branch from.  This revision will be the first
 237ancestor of the new commit.
 238
 239Omitting the `from` command in the first commit of a new branch will
 240cause gfi to create that commit with no ancestor. This tends to be
 241desired only for the initial commit of a project.  Omitting the
 242`from` command on existing branches is required, as the current
 243commit on that branch is automatically assumed to be the first
 244ancestor of the new commit.
 245
 246As `LF` is not valid in a Git refname or SHA-1 expression, no
 247quoting or escaping syntax is supported within `<committish>`.
 248
 249Here `<committish>` is any of the following:
 250
 251* The name of an existing branch already in gfi's internal branch
 252  table.  If gfi doesn't know the name, its treated as a SHA-1
 253  expression.
 254
 255* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
 256+
 257The reason gfi uses `:` to denote a mark reference is this character
 258is not legal in a Git branch name.  The leading `:` makes it easy
 259to distingush between the mark 42 (`:42`) and the branch 42 (`42`
 260or `refs/heads/42`), or an abbreviated SHA-1 which happened to
 261consist only of base-10 digits.
 262+
 263Marks must be declared (via `mark`) before they can be used.
 264
 265* A complete 40 byte or abbreviated commit SHA-1 in hex.
 266
 267* Any valid Git SHA-1 expression that resolves to a commit.  See
 268  ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
 269
 270The special case of restarting an incremental import from the
 271current branch value should be written as:
 272----
 273        from refs/heads/branch^0
 274----
 275The `^0` suffix is necessary as gfi does not permit a branch to
 276start from itself, and the branch is created in memory before the
 277`from` command is even read from the input.  Adding `^0` will force
 278gfi to resolve the commit through Git's revision parsing library,
 279rather than its internal branch table, thereby loading in the
 280existing value of the branch.
 281
 282`merge`
 283^^^^^^^
 284Includes one additional ancestor commit, and makes the current
 285commit a merge commit.  An unlimited number of `merge` commands per
 286commit are permitted by gfi, thereby establishing an n-way merge.
 287However Git's other tools never create commits with more than 15
 288additional ancestors (forming a 16-way merge).  For this reason
 289it is suggested that frontends do not use more than 15 `merge`
 290commands per commit.
 291
 292Here `<committish>` is any of the commit specification expressions
 293also accepted by `from` (see above).
 294
 295`filemodify`
 296^^^^^^^^^^
 297Included in a `commit` command to add a new file or change the
 298content of an existing file.  This command has two different means
 299of specifying the content of the file.
 300
 301External data format::
 302        The data content for the file was already supplied by a prior
 303        `blob` command.  The frontend just needs to connect it.
 304+
 305....
 306        'M' SP <mode> SP <dataref> SP <path> LF
 307....
 308+
 309Here `<dataref>` can be either a mark reference (`:<idnum>`)
 310set by a prior `blob` command, or a full 40-byte SHA-1 of an
 311existing Git blob object.
 312
 313Inline data format::
 314        The data content for the file has not been supplied yet.
 315        The frontend wants to supply it as part of this modify
 316        command.
 317+
 318....
 319        'M' SP <mode> SP 'inline' SP <path> LF
 320        data
 321....
 322+
 323See below for a detailed description of the `data` command.
 324
 325In both formats `<mode>` is the type of file entry, specified
 326in octal.  Git only supports the following modes:
 327
 328* `100644` or `644`: A normal (not-executable) file.  The majority
 329  of files in most projects use this mode.  If in doubt, this is
 330  what you want.
 331* `100755` or `755`: A normal, but executable, file.
 332* `140000`: A symlink, the content of the file will be the link target.
 333
 334In both formats `<path>` is the complete path of the file to be added
 335(if not already existing) or modified (if already existing).
 336
 337A `<path>` string must use UNIX-style directory seperators (forward
 338slash `/`), may contain any byte other than `LF`, and must not
 339start with double quote (`"`).
 340
 341If an `LF` or double quote must be encoded into `<path>` shell-style
 342quoting should be used, e.g. `"path/with\n and \" in it"`.
 343
 344The value of `<path>` must be in canoncial form. That is it must not:
 345
 346* contain an empty directory component (e.g. `foo//bar` is invalid),
 347* end with a directory seperator (e.g. `foo/` is invalid),
 348* start with a directory seperator (e.g. `/foo` is invalid),
 349* contain the special component `.` or `..` (e.g. `foo/./bar` and
 350  `foo/../bar` are invalid).
 351
 352It is recommended that `<path>` always be encoded using UTF-8.
 353
 354
 355`filedelete`
 356^^^^^^^^^^
 357Included in a `commit` command to remove a file from the branch.
 358If the file removal makes its directory empty, the directory will
 359be automatically removed too.  This cascades up the tree until the
 360first non-empty directory or the root is reached.
 361
 362....
 363        'D' SP <path> LF
 364....
 365
 366here `<path>` is the complete path of the file to be removed.
 367See `filemodify` above for a detailed description of `<path>`.
 368
 369`mark`
 370~~~~~~
 371Arranges for gfi to save a reference to the current object, allowing
 372the frontend to recall this object at a future point in time, without
 373knowing its SHA-1.  Here the current object is the object creation
 374command the `mark` command appears within.  This can be `commit`,
 375`tag`, and `blob`, but `commit` is the most common usage.
 376
 377....
 378        'mark' SP ':' <idnum> LF
 379....
 380
 381where `<idnum>` is the number assigned by the frontend to this mark.
 382The value of `<idnum>` is expressed in base 10 notation using
 383US-ASCII digits.  The value 0 is reserved and cannot be used as
 384a mark.  Only values greater than or equal to 1 may be used as marks.
 385
 386New marks are created automatically.  Existing marks can be moved
 387to another object simply by reusing the same `<idnum>` in another
 388`mark` command.
 389
 390`tag`
 391~~~~~
 392Creates an annotated tag referring to a specific commit.  To create
 393lightweight (non-annotated) tags see the `reset` command below.
 394
 395....
 396        'tag' SP <name> LF
 397        'from' SP <committish> LF
 398        'tagger' SP <name> SP LT <email> GT SP <time> SP <tz> LF
 399        data
 400        LF
 401....
 402
 403where `<name>` is the name of the tag to create.
 404
 405Tag names are automatically prefixed with `refs/tags/` when stored
 406in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
 407use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the
 408corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
 409
 410The value of `<name>` must be a valid refname in Git and therefore
 411may contain forward slashes.  As `LF` is not valid in a Git refname,
 412no quoting or escaping syntax is supported here.
 413
 414The `from` command is the same as in the `commit` command; see
 415above for details.
 416
 417The `tagger` command uses the same format as `committer` within
 418`commit`; again see above for details.
 419
 420The `data` command following `tagger` must supply the annotated tag
 421message (see below for `data` command syntax).  To import an empty
 422tag message use a 0 length data.  Tag messages are free-form and are
 423not interpreted by Git.  Currently they must be encoded in UTF-8,
 424as gfi does not permit other encodings to be specified.
 425
 426Signing annotated tags during import from within gfi is not
 427supported.  Trying to include your own PGP/GPG signature is not
 428recommended, as the frontend does not (easily) have access to the
 429complete set of bytes which normally goes into such a signature.
 430If signing is required, create lightweight tags from within gfi with
 431`reset`, then create the annotated versions of those tags offline
 432with the standard gitlink:git-tag[1] process.
 433
 434`reset`
 435~~~~~~~
 436Creates (or recreates) the named branch, optionally starting from
 437a specific revision.  The reset command allows a frontend to issue
 438a new `from` command for an existing branch, or to create a new
 439branch from an existing commit without creating a new commit.
 440
 441....
 442        'reset' SP <ref> LF
 443        ('from' SP <committish> LF)?
 444        LF
 445....
 446
 447For a detailed description of `<ref>` and `<committish>` see above
 448under `commit` and `from`.
 449
 450The `reset` command can also be used to create lightweight
 451(non-annotated) tags.  For example:
 452
 453====
 454        reset refs/tags/938
 455        from :938
 456====
 457
 458would create the lightweight tag `refs/tags/938` referring to
 459whatever commit mark `:938` references.
 460
 461`blob`
 462~~~~~~
 463Requests writing one file revision to the packfile.  The revision
 464is not connected to any commit; this connection must be formed in
 465a subsequent `commit` command by referencing the blob through an
 466assigned mark.
 467
 468....
 469        'blob' LF
 470        mark?
 471        data
 472....
 473
 474The mark command is optional here as some frontends have chosen
 475to generate the Git SHA-1 for the blob on their own, and feed that
 476directly to `commit`.  This is typically more work than its worth
 477however, as marks are inexpensive to store and easy to use.
 478
 479`data`
 480~~~~~~
 481Supplies raw data (for use as blob/file content, commit messages, or
 482annotated tag messages) to gfi.  Data can be supplied using an exact
 483byte count or delimited with a terminating line.  Real frontends
 484intended for production-quality conversions should always use the
 485exact byte count format, as it is more robust and performs better.
 486The delimited format is intended primarily for testing gfi.
 487
 488Exact byte count format:
 489
 490....
 491        'data' SP <count> LF
 492        <raw> LF
 493....
 494
 495where `<count>` is the exact number of bytes appearing within
 496`<raw>`.  The value of `<count>` is expressed in base 10 notation
 497using US-ASCII digits.  The `LF` on either side of `<raw>` is not
 498included in `<count>` and will not be included in the imported data.
 499
 500Delimited format:
 501
 502....
 503        'data' SP '<<' <delim> LF
 504        <raw> LF
 505        <delim> LF
 506....
 507
 508where `<delim>` is the chosen delimiter string.  The string `<delim>`
 509must not appear on a line by itself within `<raw>`, as otherwise
 510gfi will think the data ends earlier than it really does.  The `LF`
 511immediately trailing `<raw>` is part of `<raw>`.  This is one of
 512the limitations of the delimited format, it is impossible to supply
 513a data chunk which does not have an LF as its last byte.
 514
 515`checkpoint`
 516~~~~~~~~~~~~
 517Forces gfi to close the current packfile and start a new one.
 518As this requires a significant amount of CPU time and disk IO
 519(to compute the overall pack SHA-1 checksum and generate the
 520corresponding index file) it can easily take several minutes for
 521a single `checkpoint` command to complete.
 522
 523....
 524        'checkpoint' LF
 525        LF
 526....
 527
 528Packfile Optimization
 529---------------------
 530When packing a blob gfi always attempts to deltify against the last
 531blob written.  Unless specifically arranged for by the frontend,
 532this will probably not be a prior version of the same file, so the
 533generated delta will not be the smallest possible.  The resulting
 534packfile will be compressed, but will not be optimal.
 535
 536Frontends which have efficient access to all revisions of a
 537single file (for example reading an RCS/CVS ,v file) can choose
 538to supply all revisions of that file as a sequence of consecutive
 539`blob` commands.  This allows gfi to deltify the different file
 540revisions against each other, saving space in the final packfile.
 541Marks can be used to later identify individual file revisions during
 542a sequence of `commit` commands.
 543
 544The packfile(s) created by gfi do not encourage good disk access
 545patterns.  This is caused by gfi writing the data in the order
 546it is received on standard input, while Git typically organizes
 547data within packfiles to make the most recent (current tip) data
 548appear before historical data.  Git also clusters commits together,
 549speeding up revision traversal through better cache locality.
 550
 551For this reason it is strongly recommended that users repack the
 552repository with `git repack -a -d` after gfi completes, allowing
 553Git to reorganize the packfiles for faster data access.  If blob
 554deltas are suboptimal (see above) then also adding the `-f` option
 555to force recomputation of all deltas can significantly reduce the
 556final packfile size (30-50% smaller can be quite typical).
 557
 558Memory Utilization
 559------------------
 560There are a number of factors which affect how much memory gfi
 561requires to perform an import.  Like critical sections of core
 562Git, gfi uses its own memory allocators to ammortize any overheads
 563associated with malloc.  In practice gfi tends to ammoritize any
 564malloc overheads to 0, due to its use of large block allocations.
 565
 566per object
 567~~~~~~~~~~
 568gfi maintains an in-memory structure for every object written in
 569this execution.  On a 32 bit system the structure is 32 bytes,
 570on a 64 bit system the structure is 40 bytes (due to the larger
 571pointer sizes).  Objects in the table are not deallocated until
 572gfi terminates.  Importing 2 million objects on a 32 bit system
 573will require approximately 64 MiB of memory.
 574
 575The object table is actually a hashtable keyed on the object name
 576(the unique SHA-1).  This storage configuration allows gfi to reuse
 577an existing or already written object and avoid writing duplicates
 578to the output packfile.  Duplicate blobs are surprisingly common
 579in an import, typically due to branch merges in the source.
 580
 581per mark
 582~~~~~~~~
 583Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
 584bytes, depending on pointer size) per mark.  Although the array
 585is sparse, frontends are still strongly encouraged to use marks
 586between 1 and n, where n is the total number of marks required for
 587this import.
 588
 589per branch
 590~~~~~~~~~~
 591Branches are classified as active and inactive.  The memory usage
 592of the two classes is significantly different.
 593
 594Inactive branches are stored in a structure which uses 96 or 120
 595bytes (32 bit or 64 bit systems, respectively), plus the length of
 596the branch name (typically under 200 bytes), per branch.  gfi will
 597easily handle as many as 10,000 inactive branches in under 2 MiB
 598of memory.
 599
 600Active branches have the same overhead as inactive branches, but
 601also contain copies of every tree that has been recently modified on
 602that branch.  If subtree `include` has not been modified since the
 603branch became active, its contents will not be loaded into memory,
 604but if subtree `src` has been modified by a commit since the branch
 605became active, then its contents will be loaded in memory.
 606
 607As active branches store metadata about the files contained on that
 608branch, their in-memory storage size can grow to a considerable size
 609(see below).
 610
 611gfi automatically moves active branches to inactive status based on
 612a simple least-recently-used algorithm.  The LRU chain is updated on
 613each `commit` command.  The maximum number of active branches can be
 614increased or decreased on the command line with `--active-branches=`.
 615
 616per active tree
 617~~~~~~~~~~~~~~~
 618Trees (aka directories) use just 12 bytes of memory on top of the
 619memory required for their entries (see ``per active file'' below).
 620The cost of a tree is virtually 0, as its overhead ammortizes out
 621over the individual file entries.
 622
 623per active file entry
 624~~~~~~~~~~~~~~~~~~~~~
 625Files (and pointers to subtrees) within active trees require 52 or 64
 626bytes (32/64 bit platforms) per entry.  To conserve space, file and
 627tree names are pooled in a common string table, allowing the filename
 628``Makefile'' to use just 16 bytes (after including the string header
 629overhead) no matter how many times it occurs within the project.
 630
 631The active branch LRU, when coupled with the filename string pool
 632and lazy loading of subtrees, allows gfi to efficiently import
 633projects with 2,000+ branches and 45,114+ files in a very limited
 634memory footprint (less than 2.7 MiB per active branch).
 635
 636
 637Author
 638------
 639Written by Shawn O. Pearce <spearce@spearce.org>.
 640
 641Documentation
 642--------------
 643Documentation by Shawn O. Pearce <spearce@spearce.org>.
 644
 645GIT
 646---
 647Part of the gitlink:git[7] suite
 648