8bb5a0b36b2850de05e6a1588ad236acc82aa882
   1git-fast-import(1)
   2==================
   3
   4NAME
   5----
   6git-fast-import - Backend for fast Git data importers.
   7
   8
   9SYNOPSIS
  10--------
  11frontend | 'git-fast-import' [options]
  12
  13DESCRIPTION
  14-----------
  15This program is usually not what the end user wants to run directly.
  16Most end users want to use one of the existing frontend programs,
  17which parses a specific type of foreign source and feeds the contents
  18stored there to git-fast-import (gfi).
  19
  20gfi reads a mixed command/data stream from standard input and
  21writes one or more packfiles directly into the current repository.
  22When EOF is received on standard input, fast import writes out
  23updated branch and tag refs, fully updating the current repository
  24with the newly imported data.
  25
  26The gfi backend itself can import into an empty repository (one that
  27has already been initialized by gitlink:git-init[1]) or incrementally
  28update an existing populated repository.  Whether or not incremental
  29imports are supported from a particular foreign source depends on
  30the frontend program in use.
  31
  32
  33OPTIONS
  34-------
  35--date-format=<fmt>::
  36        Specify the type of dates the frontend will supply to
  37        gfi within `author`, `committer` and `tagger` commands.
  38        See ``Date Formats'' below for details about which formats
  39        are supported, and their syntax.
  40
  41--force::
  42        Force updating modified existing branches, even if doing
  43        so would cause commits to be lost (as the new commit does
  44        not contain the old commit).
  45
  46--max-pack-size=<n>::
  47        Maximum size of each output packfile, expressed in MiB.
  48        The default is 4096 (4 GiB) as that is the maximum allowed
  49        packfile size (due to file format limitations). Some
  50        importers may wish to lower this, such as to ensure the
  51        resulting packfiles fit on CDs.
  52
  53--depth=<n>::
  54        Maximum delta depth, for blob and tree deltification.
  55        Default is 10.
  56
  57--active-branches=<n>::
  58        Maximum number of branches to maintain active at once.
  59        See ``Memory Utilization'' below for details.  Default is 5.
  60
  61--export-marks=<file>::
  62        Dumps the internal marks table to <file> when complete.
  63        Marks are written one per line as `:markid SHA-1`.
  64        Frontends can use this file to validate imports after they
  65        have been completed.
  66
  67Performance
  68-----------
  69The design of gfi allows it to import large projects in a minimum
  70amount of memory usage and processing time.  Assuming the frontend
  71is able to keep up with gfi and feed it a constant stream of data,
  72import times for projects holding 10+ years of history and containing
  73100,000+ individual commits are generally completed in just 1-2
  74hours on quite modest (~$2,000 USD) hardware.
  75
  76Most bottlenecks appear to be in foreign source data access (the
  77source just cannot extract revisions fast enough) or disk IO (gfi
  78writes as fast as the disk will take the data).  Imports will run
  79faster if the source data is stored on a different drive than the
  80destination Git repository (due to less IO contention).
  81
  82
  83Development Cost
  84----------------
  85A typical frontend for gfi tends to weigh in at approximately 200
  86lines of Perl/Python/Ruby code.  Most developers have been able to
  87create working importers in just a couple of hours, even though it
  88is their first exposure to gfi, and sometimes even to Git.  This is
  89an ideal situation, given that most conversion tools are throw-away
  90(use once, and never look back).
  91
  92
  93Parallel Operation
  94------------------
  95Like `git-push` or `git-fetch`, imports handled by gfi are safe to
  96run alongside parallel `git repack -a -d` or `git gc` invocations,
  97or any other Git operation (including `git prune`, as loose objects
  98are never used by gfi).
  99
 100gfi does not lock the branch or tag refs it is actively importing.
 101After the import, during its ref update phase, gfi tests each
 102existing branch ref to verify the update will be a fast-forward
 103update (the commit stored in the ref is contained in the new
 104history of the commit to be written).  If the update is not a
 105fast-forward update, gfi will skip updating that ref and instead
 106prints a warning message.  gfi will always attempt to update all
 107branch refs, and does not stop on the first failure.
 108
 109Branch updates can be forced with `--force`, but its recommended that
 110this only be used on an otherwise quiet repository.  Using `--force`
 111is not necessary for an initial import into an empty repository.
 112
 113
 114Technical Discussion
 115--------------------
 116gfi tracks a set of branches in memory.  Any branch can be created
 117or modified at any point during the import process by sending a
 118`commit` command on the input stream.  This design allows a frontend
 119program to process an unlimited number of branches simultaneously,
 120generating commits in the order they are available from the source
 121data.  It also simplifies the frontend programs considerably.
 122
 123gfi does not use or alter the current working directory, or any
 124file within it.  (It does however update the current Git repository,
 125as referenced by `GIT_DIR`.)  Therefore an import frontend may use
 126the working directory for its own purposes, such as extracting file
 127revisions from the foreign source.  This ignorance of the working
 128directory also allows gfi to run very quickly, as it does not
 129need to perform any costly file update operations when switching
 130between branches.
 131
 132Input Format
 133------------
 134With the exception of raw file data (which Git does not interpret)
 135the gfi input format is text (ASCII) based.  This text based
 136format simplifies development and debugging of frontend programs,
 137especially when a higher level language such as Perl, Python or
 138Ruby is being used.
 139
 140gfi is very strict about its input.  Where we say SP below we mean
 141*exactly* one space.  Likewise LF means one (and only one) linefeed.
 142Supplying additional whitespace characters will cause unexpected
 143results, such as branch names or file names with leading or trailing
 144spaces in their name, or early termination of gfi when it encounters
 145unexpected input.
 146
 147Date Formats
 148~~~~~~~~~~~~
 149The following date formats are supported.  A frontend should select
 150the format it will use for this import by passing the format name
 151in the `--date-format=<fmt>` command line option.
 152
 153`raw`::
 154        This is the Git native format and is `<time> SP <offutc>`.
 155        It is also gfi's default format, if `--date-format` was
 156        not specified.
 157+
 158The time of the event is specified by `<time>` as the number of
 159seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
 160written as an ASCII decimal integer.
 161+
 162The local offset is specified by `<offutc>` as a positive or negative
 163offset from UTC.  For example EST (which is 5 hours behind UTC)
 164would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''.
 165The local offset does not affect `<time>`; it is used only as an
 166advisement to help formatting routines display the timestamp.
 167+
 168If the local offset is not available in the source material, use
 169``+0000'', or the most common local offset.  For example many
 170organizations have a CVS repository which has only ever been accessed
 171by users who are located in the same location and timezone.  In this
 172case the offset from UTC can be easily assumed.
 173+
 174Unlike the `rfc2822` format, this format is very strict.  Any
 175variation in formatting will cause gfi to reject the value.
 176
 177`rfc2822`::
 178        This is the standard email format as described by RFC 2822.
 179+
 180An example value is ``Tue Feb 6 11:22:18 2007 -0500''.  The Git
 181parser is accurate, but a little on the lenient side.  Its the
 182same parser used by gitlink:git-am[1] when applying patches
 183received from email.
 184+
 185Some malformed strings may be accepted as valid dates.  In some of
 186these cases Git will still be able to obtain the correct date from
 187the malformed string.  There are also some types of malformed
 188strings which Git will parse wrong, and yet consider valid.
 189Seriously malformed strings will be rejected.
 190+
 191Unlike the `raw` format above, the timezone/UTC offset information
 192contained in an RFC 2822 date string is used to adjust the date
 193value to UTC prior to storage.  Therefore it is important that
 194this information be as accurate as possible.
 195+
 196If the source material is formatted in RFC 2822 style dates,
 197the frontend should let gfi handle the parsing and conversion
 198(rather than attempting to do it itself) as the Git parser has
 199been well tested in the wild.
 200+
 201Frontends should prefer the `raw` format if the source material
 202is already in UNIX-epoch format, or is easily convertible to
 203that format, as there is no ambiguity in parsing.
 204
 205`now`::
 206        Always use the current time and timezone.  The literal
 207        `now` must always be supplied for `<when>`.
 208+
 209This is a toy format.  The current time and timezone of this system
 210is always copied into the identity string at the time it is being
 211created by gfi.  There is no way to specify a different time or
 212timezone.
 213+
 214This particular format is supplied as its short to implement and
 215may be useful to a process that wants to create a new commit
 216right now, without needing to use a working directory or
 217gitlink:git-update-index[1].
 218+
 219If separate `author` and `committer` commands are used in a `commit`
 220the timestamps may not match, as the system clock will be polled
 221twice (once for each command).  The only way to ensure that both
 222author and committer identity information has the same timestamp
 223is to omit `author` (thus copying from `committer`) or to use a
 224date format other than `now`.
 225
 226Commands
 227~~~~~~~~
 228gfi accepts several commands to update the current repository
 229and control the current import process.  More detailed discussion
 230(with examples) of each command follows later.
 231
 232`commit`::
 233        Creates a new branch or updates an existing branch by
 234        creating a new commit and updating the branch to point at
 235        the newly created commit.
 236
 237`tag`::
 238        Creates an annotated tag object from an existing commit or
 239        branch.  Lightweight tags are not supported by this command,
 240        as they are not recommended for recording meaningful points
 241        in time.
 242
 243`reset`::
 244        Reset an existing branch (or a new branch) to a specific
 245        revision.  This command must be used to change a branch to
 246        a specific revision without making a commit on it.
 247
 248`blob`::
 249        Convert raw file data into a blob, for future use in a
 250        `commit` command.  This command is optional and is not
 251        needed to perform an import.
 252
 253`checkpoint`::
 254        Forces gfi to close the current packfile, generate its
 255        unique SHA-1 checksum and index, and start a new packfile.
 256        This command is optional and is not needed to perform
 257        an import.
 258
 259`commit`
 260~~~~~~~~
 261Create or update a branch with a new commit, recording one logical
 262change to the project.
 263
 264....
 265        'commit' SP <ref> LF
 266        mark?
 267        ('author' SP <name> SP LT <email> GT SP <when> LF)?
 268        'committer' SP <name> SP LT <email> GT SP <when> LF
 269        data
 270        ('from' SP <committish> LF)?
 271        ('merge' SP <committish> LF)?
 272        (filemodify | filedelete | filedeleteall)*
 273        LF
 274....
 275
 276where `<ref>` is the name of the branch to make the commit on.
 277Typically branch names are prefixed with `refs/heads/` in
 278Git, so importing the CVS branch symbol `RELENG-1_0` would use
 279`refs/heads/RELENG-1_0` for the value of `<ref>`.  The value of
 280`<ref>` must be a valid refname in Git.  As `LF` is not valid in
 281a Git refname, no quoting or escaping syntax is supported here.
 282
 283A `mark` command may optionally appear, requesting gfi to save a
 284reference to the newly created commit for future use by the frontend
 285(see below for format).  It is very common for frontends to mark
 286every commit they create, thereby allowing future branch creation
 287from any imported commit.
 288
 289The `data` command following `committer` must supply the commit
 290message (see below for `data` command syntax).  To import an empty
 291commit message use a 0 length data.  Commit messages are free-form
 292and are not interpreted by Git.  Currently they must be encoded in
 293UTF-8, as gfi does not permit other encodings to be specified.
 294
 295Zero or more `filemodify`, `filedelete` and `filedeleteall` commands
 296may be included to update the contents of the branch prior to
 297creating the commit.  These commands may be supplied in any order.
 298However it is recommended that a `filedeleteall` command preceed
 299all `filemodify` commands in the same commit, as `filedeleteall`
 300wipes the branch clean (see below).
 301
 302`author`
 303^^^^^^^^
 304An `author` command may optionally appear, if the author information
 305might differ from the committer information.  If `author` is omitted
 306then gfi will automatically use the committer's information for
 307the author portion of the commit.  See below for a description of
 308the fields in `author`, as they are identical to `committer`.
 309
 310`committer`
 311^^^^^^^^^^^
 312The `committer` command indicates who made this commit, and when
 313they made it.
 314
 315Here `<name>` is the person's display name (for example
 316``Com M Itter'') and `<email>` is the person's email address
 317(``cm@example.com'').  `LT` and `GT` are the literal less-than (\x3c)
 318and greater-than (\x3e) symbols.  These are required to delimit
 319the email address from the other fields in the line.  Note that
 320`<name>` is free-form and may contain any sequence of bytes, except
 321`LT` and `LF`.  It is typically UTF-8 encoded.
 322
 323The time of the change is specified by `<when>` using the date format
 324that was selected by the `--date-format=<fmt>` command line option.
 325See ``Date Formats'' above for the set of supported formats, and
 326their syntax.
 327
 328`from`
 329^^^^^^
 330Only valid for the first commit made on this branch by this
 331gfi process.  The `from` command is used to specify the commit
 332to initialize this branch from.  This revision will be the first
 333ancestor of the new commit.
 334
 335Omitting the `from` command in the first commit of a new branch will
 336cause gfi to create that commit with no ancestor. This tends to be
 337desired only for the initial commit of a project.  Omitting the
 338`from` command on existing branches is required, as the current
 339commit on that branch is automatically assumed to be the first
 340ancestor of the new commit.
 341
 342As `LF` is not valid in a Git refname or SHA-1 expression, no
 343quoting or escaping syntax is supported within `<committish>`.
 344
 345Here `<committish>` is any of the following:
 346
 347* The name of an existing branch already in gfi's internal branch
 348  table.  If gfi doesn't know the name, its treated as a SHA-1
 349  expression.
 350
 351* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
 352+
 353The reason gfi uses `:` to denote a mark reference is this character
 354is not legal in a Git branch name.  The leading `:` makes it easy
 355to distingush between the mark 42 (`:42`) and the branch 42 (`42`
 356or `refs/heads/42`), or an abbreviated SHA-1 which happened to
 357consist only of base-10 digits.
 358+
 359Marks must be declared (via `mark`) before they can be used.
 360
 361* A complete 40 byte or abbreviated commit SHA-1 in hex.
 362
 363* Any valid Git SHA-1 expression that resolves to a commit.  See
 364  ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
 365
 366The special case of restarting an incremental import from the
 367current branch value should be written as:
 368----
 369        from refs/heads/branch^0
 370----
 371The `^0` suffix is necessary as gfi does not permit a branch to
 372start from itself, and the branch is created in memory before the
 373`from` command is even read from the input.  Adding `^0` will force
 374gfi to resolve the commit through Git's revision parsing library,
 375rather than its internal branch table, thereby loading in the
 376existing value of the branch.
 377
 378`merge`
 379^^^^^^^
 380Includes one additional ancestor commit, and makes the current
 381commit a merge commit.  An unlimited number of `merge` commands per
 382commit are permitted by gfi, thereby establishing an n-way merge.
 383However Git's other tools never create commits with more than 15
 384additional ancestors (forming a 16-way merge).  For this reason
 385it is suggested that frontends do not use more than 15 `merge`
 386commands per commit.
 387
 388Here `<committish>` is any of the commit specification expressions
 389also accepted by `from` (see above).
 390
 391`filemodify`
 392^^^^^^^^^^^^
 393Included in a `commit` command to add a new file or change the
 394content of an existing file.  This command has two different means
 395of specifying the content of the file.
 396
 397External data format::
 398        The data content for the file was already supplied by a prior
 399        `blob` command.  The frontend just needs to connect it.
 400+
 401....
 402        'M' SP <mode> SP <dataref> SP <path> LF
 403....
 404+
 405Here `<dataref>` can be either a mark reference (`:<idnum>`)
 406set by a prior `blob` command, or a full 40-byte SHA-1 of an
 407existing Git blob object.
 408
 409Inline data format::
 410        The data content for the file has not been supplied yet.
 411        The frontend wants to supply it as part of this modify
 412        command.
 413+
 414....
 415        'M' SP <mode> SP 'inline' SP <path> LF
 416        data
 417....
 418+
 419See below for a detailed description of the `data` command.
 420
 421In both formats `<mode>` is the type of file entry, specified
 422in octal.  Git only supports the following modes:
 423
 424* `100644` or `644`: A normal (not-executable) file.  The majority
 425  of files in most projects use this mode.  If in doubt, this is
 426  what you want.
 427* `100755` or `755`: A normal, but executable, file.
 428* `120000`: A symlink, the content of the file will be the link target.
 429
 430In both formats `<path>` is the complete path of the file to be added
 431(if not already existing) or modified (if already existing).
 432
 433A `<path>` string must use UNIX-style directory seperators (forward
 434slash `/`), may contain any byte other than `LF`, and must not
 435start with double quote (`"`).
 436
 437If an `LF` or double quote must be encoded into `<path>` shell-style
 438quoting should be used, e.g. `"path/with\n and \" in it"`.
 439
 440The value of `<path>` must be in canoncial form. That is it must not:
 441
 442* contain an empty directory component (e.g. `foo//bar` is invalid),
 443* end with a directory seperator (e.g. `foo/` is invalid),
 444* start with a directory seperator (e.g. `/foo` is invalid),
 445* contain the special component `.` or `..` (e.g. `foo/./bar` and
 446  `foo/../bar` are invalid).
 447
 448It is recommended that `<path>` always be encoded using UTF-8.
 449
 450`filedelete`
 451^^^^^^^^^^^^
 452Included in a `commit` command to remove a file from the branch.
 453If the file removal makes its directory empty, the directory will
 454be automatically removed too.  This cascades up the tree until the
 455first non-empty directory or the root is reached.
 456
 457....
 458        'D' SP <path> LF
 459....
 460
 461here `<path>` is the complete path of the file to be removed.
 462See `filemodify` above for a detailed description of `<path>`.
 463
 464`filedeleteall`
 465^^^^^^^^^^^^^^^
 466Included in a `commit` command to remove all files (and also all
 467directories) from the branch.  This command resets the internal
 468branch structure to have no files in it, allowing the frontend
 469to subsequently add all interesting files from scratch.
 470
 471....
 472        'deleteall' LF
 473....
 474
 475This command is extremely useful if the frontend does not know
 476(or does not care to know) what files are currently on the branch,
 477and therefore cannot generate the proper `filedelete` commands to
 478update the content.
 479
 480Issuing a `filedeleteall` followed by the needed `filemodify`
 481commands to set the correct content will produce the same results
 482as sending only the needed `filemodify` and `filedelete` commands.
 483The `filedeleteall` approach may however require gfi to use slightly
 484more memory per active branch (less than 1 MiB for even most large
 485projects); so frontends that can easily obtain only the affected
 486paths for a commit are encouraged to do so.
 487
 488`mark`
 489~~~~~~
 490Arranges for gfi to save a reference to the current object, allowing
 491the frontend to recall this object at a future point in time, without
 492knowing its SHA-1.  Here the current object is the object creation
 493command the `mark` command appears within.  This can be `commit`,
 494`tag`, and `blob`, but `commit` is the most common usage.
 495
 496....
 497        'mark' SP ':' <idnum> LF
 498....
 499
 500where `<idnum>` is the number assigned by the frontend to this mark.
 501The value of `<idnum>` is expressed as an ASCII decimal integer.
 502The value 0 is reserved and cannot be used as
 503a mark.  Only values greater than or equal to 1 may be used as marks.
 504
 505New marks are created automatically.  Existing marks can be moved
 506to another object simply by reusing the same `<idnum>` in another
 507`mark` command.
 508
 509`tag`
 510~~~~~
 511Creates an annotated tag referring to a specific commit.  To create
 512lightweight (non-annotated) tags see the `reset` command below.
 513
 514....
 515        'tag' SP <name> LF
 516        'from' SP <committish> LF
 517        'tagger' SP <name> SP LT <email> GT SP <when> LF
 518        data
 519        LF
 520....
 521
 522where `<name>` is the name of the tag to create.
 523
 524Tag names are automatically prefixed with `refs/tags/` when stored
 525in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
 526use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the
 527corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
 528
 529The value of `<name>` must be a valid refname in Git and therefore
 530may contain forward slashes.  As `LF` is not valid in a Git refname,
 531no quoting or escaping syntax is supported here.
 532
 533The `from` command is the same as in the `commit` command; see
 534above for details.
 535
 536The `tagger` command uses the same format as `committer` within
 537`commit`; again see above for details.
 538
 539The `data` command following `tagger` must supply the annotated tag
 540message (see below for `data` command syntax).  To import an empty
 541tag message use a 0 length data.  Tag messages are free-form and are
 542not interpreted by Git.  Currently they must be encoded in UTF-8,
 543as gfi does not permit other encodings to be specified.
 544
 545Signing annotated tags during import from within gfi is not
 546supported.  Trying to include your own PGP/GPG signature is not
 547recommended, as the frontend does not (easily) have access to the
 548complete set of bytes which normally goes into such a signature.
 549If signing is required, create lightweight tags from within gfi with
 550`reset`, then create the annotated versions of those tags offline
 551with the standard gitlink:git-tag[1] process.
 552
 553`reset`
 554~~~~~~~
 555Creates (or recreates) the named branch, optionally starting from
 556a specific revision.  The reset command allows a frontend to issue
 557a new `from` command for an existing branch, or to create a new
 558branch from an existing commit without creating a new commit.
 559
 560....
 561        'reset' SP <ref> LF
 562        ('from' SP <committish> LF)?
 563        LF
 564....
 565
 566For a detailed description of `<ref>` and `<committish>` see above
 567under `commit` and `from`.
 568
 569The `reset` command can also be used to create lightweight
 570(non-annotated) tags.  For example:
 571
 572====
 573        reset refs/tags/938
 574        from :938
 575====
 576
 577would create the lightweight tag `refs/tags/938` referring to
 578whatever commit mark `:938` references.
 579
 580`blob`
 581~~~~~~
 582Requests writing one file revision to the packfile.  The revision
 583is not connected to any commit; this connection must be formed in
 584a subsequent `commit` command by referencing the blob through an
 585assigned mark.
 586
 587....
 588        'blob' LF
 589        mark?
 590        data
 591....
 592
 593The mark command is optional here as some frontends have chosen
 594to generate the Git SHA-1 for the blob on their own, and feed that
 595directly to `commit`.  This is typically more work than its worth
 596however, as marks are inexpensive to store and easy to use.
 597
 598`data`
 599~~~~~~
 600Supplies raw data (for use as blob/file content, commit messages, or
 601annotated tag messages) to gfi.  Data can be supplied using an exact
 602byte count or delimited with a terminating line.  Real frontends
 603intended for production-quality conversions should always use the
 604exact byte count format, as it is more robust and performs better.
 605The delimited format is intended primarily for testing gfi.
 606
 607Exact byte count format::
 608        The frontend must specify the number of bytes of data.
 609+
 610....
 611        'data' SP <count> LF
 612        <raw> LF
 613....
 614+
 615where `<count>` is the exact number of bytes appearing within
 616`<raw>`.  The value of `<count>` is expressed as an ASCII decimal
 617integer.  The `LF` on either side of `<raw>` is not
 618included in `<count>` and will not be included in the imported data.
 619
 620Delimited format::
 621        A delimiter string is used to mark the end of the data.
 622        gfi will compute the length by searching for the delimiter.
 623        This format is primarly useful for testing and is not
 624        recommended for real data.
 625+
 626....
 627        'data' SP '<<' <delim> LF
 628        <raw> LF
 629        <delim> LF
 630....
 631+
 632where `<delim>` is the chosen delimiter string.  The string `<delim>`
 633must not appear on a line by itself within `<raw>`, as otherwise
 634gfi will think the data ends earlier than it really does.  The `LF`
 635immediately trailing `<raw>` is part of `<raw>`.  This is one of
 636the limitations of the delimited format, it is impossible to supply
 637a data chunk which does not have an LF as its last byte.
 638
 639`checkpoint`
 640~~~~~~~~~~~~
 641Forces gfi to close the current packfile and start a new one.
 642As this requires a significant amount of CPU time and disk IO
 643(to compute the overall pack SHA-1 checksum and generate the
 644corresponding index file) it can easily take several minutes for
 645a single `checkpoint` command to complete.
 646
 647....
 648        'checkpoint' LF
 649        LF
 650....
 651
 652Packfile Optimization
 653---------------------
 654When packing a blob gfi always attempts to deltify against the last
 655blob written.  Unless specifically arranged for by the frontend,
 656this will probably not be a prior version of the same file, so the
 657generated delta will not be the smallest possible.  The resulting
 658packfile will be compressed, but will not be optimal.
 659
 660Frontends which have efficient access to all revisions of a
 661single file (for example reading an RCS/CVS ,v file) can choose
 662to supply all revisions of that file as a sequence of consecutive
 663`blob` commands.  This allows gfi to deltify the different file
 664revisions against each other, saving space in the final packfile.
 665Marks can be used to later identify individual file revisions during
 666a sequence of `commit` commands.
 667
 668The packfile(s) created by gfi do not encourage good disk access
 669patterns.  This is caused by gfi writing the data in the order
 670it is received on standard input, while Git typically organizes
 671data within packfiles to make the most recent (current tip) data
 672appear before historical data.  Git also clusters commits together,
 673speeding up revision traversal through better cache locality.
 674
 675For this reason it is strongly recommended that users repack the
 676repository with `git repack -a -d` after gfi completes, allowing
 677Git to reorganize the packfiles for faster data access.  If blob
 678deltas are suboptimal (see above) then also adding the `-f` option
 679to force recomputation of all deltas can significantly reduce the
 680final packfile size (30-50% smaller can be quite typical).
 681
 682Memory Utilization
 683------------------
 684There are a number of factors which affect how much memory gfi
 685requires to perform an import.  Like critical sections of core
 686Git, gfi uses its own memory allocators to ammortize any overheads
 687associated with malloc.  In practice gfi tends to ammoritize any
 688malloc overheads to 0, due to its use of large block allocations.
 689
 690per object
 691~~~~~~~~~~
 692gfi maintains an in-memory structure for every object written in
 693this execution.  On a 32 bit system the structure is 32 bytes,
 694on a 64 bit system the structure is 40 bytes (due to the larger
 695pointer sizes).  Objects in the table are not deallocated until
 696gfi terminates.  Importing 2 million objects on a 32 bit system
 697will require approximately 64 MiB of memory.
 698
 699The object table is actually a hashtable keyed on the object name
 700(the unique SHA-1).  This storage configuration allows gfi to reuse
 701an existing or already written object and avoid writing duplicates
 702to the output packfile.  Duplicate blobs are surprisingly common
 703in an import, typically due to branch merges in the source.
 704
 705per mark
 706~~~~~~~~
 707Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
 708bytes, depending on pointer size) per mark.  Although the array
 709is sparse, frontends are still strongly encouraged to use marks
 710between 1 and n, where n is the total number of marks required for
 711this import.
 712
 713per branch
 714~~~~~~~~~~
 715Branches are classified as active and inactive.  The memory usage
 716of the two classes is significantly different.
 717
 718Inactive branches are stored in a structure which uses 96 or 120
 719bytes (32 bit or 64 bit systems, respectively), plus the length of
 720the branch name (typically under 200 bytes), per branch.  gfi will
 721easily handle as many as 10,000 inactive branches in under 2 MiB
 722of memory.
 723
 724Active branches have the same overhead as inactive branches, but
 725also contain copies of every tree that has been recently modified on
 726that branch.  If subtree `include` has not been modified since the
 727branch became active, its contents will not be loaded into memory,
 728but if subtree `src` has been modified by a commit since the branch
 729became active, then its contents will be loaded in memory.
 730
 731As active branches store metadata about the files contained on that
 732branch, their in-memory storage size can grow to a considerable size
 733(see below).
 734
 735gfi automatically moves active branches to inactive status based on
 736a simple least-recently-used algorithm.  The LRU chain is updated on
 737each `commit` command.  The maximum number of active branches can be
 738increased or decreased on the command line with `--active-branches=`.
 739
 740per active tree
 741~~~~~~~~~~~~~~~
 742Trees (aka directories) use just 12 bytes of memory on top of the
 743memory required for their entries (see ``per active file'' below).
 744The cost of a tree is virtually 0, as its overhead ammortizes out
 745over the individual file entries.
 746
 747per active file entry
 748~~~~~~~~~~~~~~~~~~~~~
 749Files (and pointers to subtrees) within active trees require 52 or 64
 750bytes (32/64 bit platforms) per entry.  To conserve space, file and
 751tree names are pooled in a common string table, allowing the filename
 752``Makefile'' to use just 16 bytes (after including the string header
 753overhead) no matter how many times it occurs within the project.
 754
 755The active branch LRU, when coupled with the filename string pool
 756and lazy loading of subtrees, allows gfi to efficiently import
 757projects with 2,000+ branches and 45,114+ files in a very limited
 758memory footprint (less than 2.7 MiB per active branch).
 759
 760
 761Author
 762------
 763Written by Shawn O. Pearce <spearce@spearce.org>.
 764
 765Documentation
 766--------------
 767Documentation by Shawn O. Pearce <spearce@spearce.org>.
 768
 769GIT
 770---
 771Part of the gitlink:git[7] suite
 772