1fe2c1dcf225d430dc8517c47cc5d48073cd96b3
   1git-fast-import(1)
   2==================
   3
   4NAME
   5----
   6git-fast-import - Backend for fast Git data importers.
   7
   8
   9SYNOPSIS
  10--------
  11frontend | 'git-fast-import' [options]
  12
  13DESCRIPTION
  14-----------
  15This program is usually not what the end user wants to run directly.
  16Most end users want to use one of the existing frontend programs,
  17which parses a specific type of foreign source and feeds the contents
  18stored there to git-fast-import (gfi).
  19
  20gfi reads a mixed command/data stream from standard input and
  21writes one or more packfiles directly into the current repository.
  22When EOF is received on standard input, fast import writes out
  23updated branch and tag refs, fully updating the current repository
  24with the newly imported data.
  25
  26The gfi backend itself can import into an empty repository (one that
  27has already been initialized by gitlink:git-init[1]) or incrementally
  28update an existing populated repository.  Whether or not incremental
  29imports are supported from a particular foreign source depends on
  30the frontend program in use.
  31
  32
  33OPTIONS
  34-------
  35--date-format=<fmt>::
  36        Specify the type of dates the frontend will supply to
  37        gfi within `author`, `committer` and `tagger` commands.
  38        See ``Date Formats'' below for details about which formats
  39        are supported, and their syntax.
  40
  41--force::
  42        Force updating modified existing branches, even if doing
  43        so would cause commits to be lost (as the new commit does
  44        not contain the old commit).
  45
  46--max-pack-size=<n>::
  47        Maximum size of each output packfile, expressed in MiB.
  48        The default is 4096 (4 GiB) as that is the maximum allowed
  49        packfile size (due to file format limitations). Some
  50        importers may wish to lower this, such as to ensure the
  51        resulting packfiles fit on CDs.
  52
  53--depth=<n>::
  54        Maximum delta depth, for blob and tree deltification.
  55        Default is 10.
  56
  57--active-branches=<n>::
  58        Maximum number of branches to maintain active at once.
  59        See ``Memory Utilization'' below for details.  Default is 5.
  60
  61--export-marks=<file>::
  62        Dumps the internal marks table to <file> when complete.
  63        Marks are written one per line as `:markid SHA-1`.
  64        Frontends can use this file to validate imports after they
  65        have been completed.
  66
  67Performance
  68-----------
  69The design of gfi allows it to import large projects in a minimum
  70amount of memory usage and processing time.  Assuming the frontend
  71is able to keep up with gfi and feed it a constant stream of data,
  72import times for projects holding 10+ years of history and containing
  73100,000+ individual commits are generally completed in just 1-2
  74hours on quite modest (~$2,000 USD) hardware.
  75
  76Most bottlenecks appear to be in foreign source data access (the
  77source just cannot extract revisions fast enough) or disk IO (gfi
  78writes as fast as the disk will take the data).  Imports will run
  79faster if the source data is stored on a different drive than the
  80destination Git repository (due to less IO contention).
  81
  82
  83Development Cost
  84----------------
  85A typical frontend for gfi tends to weigh in at approximately 200
  86lines of Perl/Python/Ruby code.  Most developers have been able to
  87create working importers in just a couple of hours, even though it
  88is their first exposure to gfi, and sometimes even to Git.  This is
  89an ideal situation, given that most conversion tools are throw-away
  90(use once, and never look back).
  91
  92
  93Parallel Operation
  94------------------
  95Like `git-push` or `git-fetch`, imports handled by gfi are safe to
  96run alongside parallel `git repack -a -d` or `git gc` invocations,
  97or any other Git operation (including `git prune`, as loose objects
  98are never used by gfi).
  99
 100gfi does not lock the branch or tag refs it is actively importing.
 101After the import, during its ref update phase, gfi tests each
 102existing branch ref to verify the update will be a fast-forward
 103update (the commit stored in the ref is contained in the new
 104history of the commit to be written).  If the update is not a
 105fast-forward update, gfi will skip updating that ref and instead
 106prints a warning message.  gfi will always attempt to update all
 107branch refs, and does not stop on the first failure.
 108
 109Branch updates can be forced with `--force`, but its recommended that
 110this only be used on an otherwise quiet repository.  Using `--force`
 111is not necessary for an initial import into an empty repository.
 112
 113
 114Technical Discussion
 115--------------------
 116gfi tracks a set of branches in memory.  Any branch can be created
 117or modified at any point during the import process by sending a
 118`commit` command on the input stream.  This design allows a frontend
 119program to process an unlimited number of branches simultaneously,
 120generating commits in the order they are available from the source
 121data.  It also simplifies the frontend programs considerably.
 122
 123gfi does not use or alter the current working directory, or any
 124file within it.  (It does however update the current Git repository,
 125as referenced by `GIT_DIR`.)  Therefore an import frontend may use
 126the working directory for its own purposes, such as extracting file
 127revisions from the foreign source.  This ignorance of the working
 128directory also allows gfi to run very quickly, as it does not
 129need to perform any costly file update operations when switching
 130between branches.
 131
 132Input Format
 133------------
 134With the exception of raw file data (which Git does not interpret)
 135the gfi input format is text (ASCII) based.  This text based
 136format simplifies development and debugging of frontend programs,
 137especially when a higher level language such as Perl, Python or
 138Ruby is being used.
 139
 140gfi is very strict about its input.  Where we say SP below we mean
 141*exactly* one space.  Likewise LF means one (and only one) linefeed.
 142Supplying additional whitespace characters will cause unexpected
 143results, such as branch names or file names with leading or trailing
 144spaces in their name, or early termination of gfi when it encounters
 145unexpected input.
 146
 147Date Formats
 148~~~~~~~~~~~~
 149The following date formats are supported.  A frontend should select
 150the format it will use for this import by passing the format name
 151in the `--date-format=<fmt>` command line option.
 152
 153`raw`::
 154        This is the Git native format and is `<time> SP <tz>`.
 155        It is also gfi's default format, if `--date-format` was
 156        not specified.
 157+
 158The time of the event is specified by `<time>` as the number of
 159seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
 160written as an ASCII decimal integer.
 161+
 162The timezone is specified by `<tz>` as a positive or negative offset
 163from UTC.  For example EST (which is typically 5 hours behind GMT)
 164would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
 165+
 166If the timezone is not available in the source material, use
 167``+0000'', or the most common local timezone.  For example many
 168organizations have a CVS repository which has only ever been accessed
 169by users who are located in the same location and timezone.  In this
 170case the user's timezone can be easily assumed.
 171+
 172Unlike the `rfc2822` format, this format is very strict.  Any
 173variation in formatting will cause gfi to reject the value.
 174
 175`rfc2822`::
 176        This is the standard email format as described by RFC 2822.
 177+
 178An example value is ``Tue Feb 6 11:22:18 2007 -0500''.  The Git
 179parser is accurate, but a little on the lenient side.  Its the
 180same parser used by gitlink:git-am[1] when applying patches
 181received from email.
 182+
 183Some malformed strings may be accepted as valid dates.  In some of
 184these cases Git will still be able to obtain the correct date from
 185the malformed string.  There are also some types of malformed
 186strings which Git will parse wrong, and yet consider valid.
 187Seriously malformed strings will be rejected.
 188+
 189If the source material is formatted in RFC 2822 style dates,
 190the frontend should let gfi handle the parsing and conversion
 191(rather than attempting to do it itself) as the Git parser has
 192been well tested in the wild.
 193+
 194Frontends should prefer the `raw` format if the source material
 195is already in UNIX-epoch format, or is easily convertible to
 196that format, as there is no ambiguity in parsing.
 197
 198`now`::
 199        Always use the current time and timezone.  The literal
 200        `now` must always be supplied for `<when>`.
 201+
 202This is a toy format.  The current time and timezone of this system
 203is always copied into the identity string at the time it is being
 204created by gfi.  There is no way to specify a different time or
 205timezone.
 206+
 207This particular format is supplied as its short to implement and
 208may be useful to a process that wants to create a new commit
 209right now, without needing to use a working directory or
 210gitlink:git-update-index[1].
 211+
 212If separate `author` and `committer` commands are used in a `commit`
 213the timestamps may not match, as the system clock will be polled
 214twice (once for each command).  The only way to ensure that both
 215author and committer identity information has the same timestamp
 216is to omit `author` (thus copying from `committer`) or to use a
 217date format other than `now`.
 218
 219Commands
 220~~~~~~~~
 221gfi accepts several commands to update the current repository
 222and control the current import process.  More detailed discussion
 223(with examples) of each command follows later.
 224
 225`commit`::
 226        Creates a new branch or updates an existing branch by
 227        creating a new commit and updating the branch to point at
 228        the newly created commit.
 229
 230`tag`::
 231        Creates an annotated tag object from an existing commit or
 232        branch.  Lightweight tags are not supported by this command,
 233        as they are not recommended for recording meaningful points
 234        in time.
 235
 236`reset`::
 237        Reset an existing branch (or a new branch) to a specific
 238        revision.  This command must be used to change a branch to
 239        a specific revision without making a commit on it.
 240
 241`blob`::
 242        Convert raw file data into a blob, for future use in a
 243        `commit` command.  This command is optional and is not
 244        needed to perform an import.
 245
 246`checkpoint`::
 247        Forces gfi to close the current packfile, generate its
 248        unique SHA-1 checksum and index, and start a new packfile.
 249        This command is optional and is not needed to perform
 250        an import.
 251
 252`commit`
 253~~~~~~~~
 254Create or update a branch with a new commit, recording one logical
 255change to the project.
 256
 257....
 258        'commit' SP <ref> LF
 259        mark?
 260        ('author' SP <name> SP LT <email> GT SP <when> LF)?
 261        'committer' SP <name> SP LT <email> GT SP <when> LF
 262        data
 263        ('from' SP <committish> LF)?
 264        ('merge' SP <committish> LF)?
 265        (filemodify | filedelete)*
 266        LF
 267....
 268
 269where `<ref>` is the name of the branch to make the commit on.
 270Typically branch names are prefixed with `refs/heads/` in
 271Git, so importing the CVS branch symbol `RELENG-1_0` would use
 272`refs/heads/RELENG-1_0` for the value of `<ref>`.  The value of
 273`<ref>` must be a valid refname in Git.  As `LF` is not valid in
 274a Git refname, no quoting or escaping syntax is supported here.
 275
 276A `mark` command may optionally appear, requesting gfi to save a
 277reference to the newly created commit for future use by the frontend
 278(see below for format).  It is very common for frontends to mark
 279every commit they create, thereby allowing future branch creation
 280from any imported commit.
 281
 282The `data` command following `committer` must supply the commit
 283message (see below for `data` command syntax).  To import an empty
 284commit message use a 0 length data.  Commit messages are free-form
 285and are not interpreted by Git.  Currently they must be encoded in
 286UTF-8, as gfi does not permit other encodings to be specified.
 287
 288Zero or more `filemodify` and `filedelete` commands may be
 289included to update the contents of the branch prior to the commit.
 290These commands can be supplied in any order, gfi is not sensitive
 291to pathname or operation ordering.
 292
 293`author`
 294^^^^^^^^
 295An `author` command may optionally appear, if the author information
 296might differ from the committer information.  If `author` is omitted
 297then gfi will automatically use the committer's information for
 298the author portion of the commit.  See below for a description of
 299the fields in `author`, as they are identical to `committer`.
 300
 301`committer`
 302^^^^^^^^^^^
 303The `committer` command indicates who made this commit, and when
 304they made it.
 305
 306Here `<name>` is the person's display name (for example
 307``Com M Itter'') and `<email>` is the person's email address
 308(``cm@example.com'').  `LT` and `GT` are the literal less-than (\x3c)
 309and greater-than (\x3e) symbols.  These are required to delimit
 310the email address from the other fields in the line.  Note that
 311`<name>` is free-form and may contain any sequence of bytes, except
 312`LT` and `LF`.  It is typically UTF-8 encoded.
 313
 314The time of the change is specified by `<when>` using the date format
 315that was selected by the `--date-format=<fmt>` command line option.
 316See ``Date Formats'' above for the set of supported formats, and
 317their syntax.
 318
 319`from`
 320^^^^^^
 321Only valid for the first commit made on this branch by this
 322gfi process.  The `from` command is used to specify the commit
 323to initialize this branch from.  This revision will be the first
 324ancestor of the new commit.
 325
 326Omitting the `from` command in the first commit of a new branch will
 327cause gfi to create that commit with no ancestor. This tends to be
 328desired only for the initial commit of a project.  Omitting the
 329`from` command on existing branches is required, as the current
 330commit on that branch is automatically assumed to be the first
 331ancestor of the new commit.
 332
 333As `LF` is not valid in a Git refname or SHA-1 expression, no
 334quoting or escaping syntax is supported within `<committish>`.
 335
 336Here `<committish>` is any of the following:
 337
 338* The name of an existing branch already in gfi's internal branch
 339  table.  If gfi doesn't know the name, its treated as a SHA-1
 340  expression.
 341
 342* A mark reference, `:<idnum>`, where `<idnum>` is the mark number.
 343+
 344The reason gfi uses `:` to denote a mark reference is this character
 345is not legal in a Git branch name.  The leading `:` makes it easy
 346to distingush between the mark 42 (`:42`) and the branch 42 (`42`
 347or `refs/heads/42`), or an abbreviated SHA-1 which happened to
 348consist only of base-10 digits.
 349+
 350Marks must be declared (via `mark`) before they can be used.
 351
 352* A complete 40 byte or abbreviated commit SHA-1 in hex.
 353
 354* Any valid Git SHA-1 expression that resolves to a commit.  See
 355  ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details.
 356
 357The special case of restarting an incremental import from the
 358current branch value should be written as:
 359----
 360        from refs/heads/branch^0
 361----
 362The `^0` suffix is necessary as gfi does not permit a branch to
 363start from itself, and the branch is created in memory before the
 364`from` command is even read from the input.  Adding `^0` will force
 365gfi to resolve the commit through Git's revision parsing library,
 366rather than its internal branch table, thereby loading in the
 367existing value of the branch.
 368
 369`merge`
 370^^^^^^^
 371Includes one additional ancestor commit, and makes the current
 372commit a merge commit.  An unlimited number of `merge` commands per
 373commit are permitted by gfi, thereby establishing an n-way merge.
 374However Git's other tools never create commits with more than 15
 375additional ancestors (forming a 16-way merge).  For this reason
 376it is suggested that frontends do not use more than 15 `merge`
 377commands per commit.
 378
 379Here `<committish>` is any of the commit specification expressions
 380also accepted by `from` (see above).
 381
 382`filemodify`
 383^^^^^^^^^^^^
 384Included in a `commit` command to add a new file or change the
 385content of an existing file.  This command has two different means
 386of specifying the content of the file.
 387
 388External data format::
 389        The data content for the file was already supplied by a prior
 390        `blob` command.  The frontend just needs to connect it.
 391+
 392....
 393        'M' SP <mode> SP <dataref> SP <path> LF
 394....
 395+
 396Here `<dataref>` can be either a mark reference (`:<idnum>`)
 397set by a prior `blob` command, or a full 40-byte SHA-1 of an
 398existing Git blob object.
 399
 400Inline data format::
 401        The data content for the file has not been supplied yet.
 402        The frontend wants to supply it as part of this modify
 403        command.
 404+
 405....
 406        'M' SP <mode> SP 'inline' SP <path> LF
 407        data
 408....
 409+
 410See below for a detailed description of the `data` command.
 411
 412In both formats `<mode>` is the type of file entry, specified
 413in octal.  Git only supports the following modes:
 414
 415* `100644` or `644`: A normal (not-executable) file.  The majority
 416  of files in most projects use this mode.  If in doubt, this is
 417  what you want.
 418* `100755` or `755`: A normal, but executable, file.
 419* `120000`: A symlink, the content of the file will be the link target.
 420
 421In both formats `<path>` is the complete path of the file to be added
 422(if not already existing) or modified (if already existing).
 423
 424A `<path>` string must use UNIX-style directory seperators (forward
 425slash `/`), may contain any byte other than `LF`, and must not
 426start with double quote (`"`).
 427
 428If an `LF` or double quote must be encoded into `<path>` shell-style
 429quoting should be used, e.g. `"path/with\n and \" in it"`.
 430
 431The value of `<path>` must be in canoncial form. That is it must not:
 432
 433* contain an empty directory component (e.g. `foo//bar` is invalid),
 434* end with a directory seperator (e.g. `foo/` is invalid),
 435* start with a directory seperator (e.g. `/foo` is invalid),
 436* contain the special component `.` or `..` (e.g. `foo/./bar` and
 437  `foo/../bar` are invalid).
 438
 439It is recommended that `<path>` always be encoded using UTF-8.
 440
 441`filedelete`
 442^^^^^^^^^^^^
 443Included in a `commit` command to remove a file from the branch.
 444If the file removal makes its directory empty, the directory will
 445be automatically removed too.  This cascades up the tree until the
 446first non-empty directory or the root is reached.
 447
 448....
 449        'D' SP <path> LF
 450....
 451
 452here `<path>` is the complete path of the file to be removed.
 453See `filemodify` above for a detailed description of `<path>`.
 454
 455`mark`
 456~~~~~~
 457Arranges for gfi to save a reference to the current object, allowing
 458the frontend to recall this object at a future point in time, without
 459knowing its SHA-1.  Here the current object is the object creation
 460command the `mark` command appears within.  This can be `commit`,
 461`tag`, and `blob`, but `commit` is the most common usage.
 462
 463....
 464        'mark' SP ':' <idnum> LF
 465....
 466
 467where `<idnum>` is the number assigned by the frontend to this mark.
 468The value of `<idnum>` is expressed as an ASCII decimal integer.
 469The value 0 is reserved and cannot be used as
 470a mark.  Only values greater than or equal to 1 may be used as marks.
 471
 472New marks are created automatically.  Existing marks can be moved
 473to another object simply by reusing the same `<idnum>` in another
 474`mark` command.
 475
 476`tag`
 477~~~~~
 478Creates an annotated tag referring to a specific commit.  To create
 479lightweight (non-annotated) tags see the `reset` command below.
 480
 481....
 482        'tag' SP <name> LF
 483        'from' SP <committish> LF
 484        'tagger' SP <name> SP LT <email> GT SP <when> LF
 485        data
 486        LF
 487....
 488
 489where `<name>` is the name of the tag to create.
 490
 491Tag names are automatically prefixed with `refs/tags/` when stored
 492in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would
 493use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the
 494corresponding ref as `refs/tags/RELENG-1_0-FINAL`.
 495
 496The value of `<name>` must be a valid refname in Git and therefore
 497may contain forward slashes.  As `LF` is not valid in a Git refname,
 498no quoting or escaping syntax is supported here.
 499
 500The `from` command is the same as in the `commit` command; see
 501above for details.
 502
 503The `tagger` command uses the same format as `committer` within
 504`commit`; again see above for details.
 505
 506The `data` command following `tagger` must supply the annotated tag
 507message (see below for `data` command syntax).  To import an empty
 508tag message use a 0 length data.  Tag messages are free-form and are
 509not interpreted by Git.  Currently they must be encoded in UTF-8,
 510as gfi does not permit other encodings to be specified.
 511
 512Signing annotated tags during import from within gfi is not
 513supported.  Trying to include your own PGP/GPG signature is not
 514recommended, as the frontend does not (easily) have access to the
 515complete set of bytes which normally goes into such a signature.
 516If signing is required, create lightweight tags from within gfi with
 517`reset`, then create the annotated versions of those tags offline
 518with the standard gitlink:git-tag[1] process.
 519
 520`reset`
 521~~~~~~~
 522Creates (or recreates) the named branch, optionally starting from
 523a specific revision.  The reset command allows a frontend to issue
 524a new `from` command for an existing branch, or to create a new
 525branch from an existing commit without creating a new commit.
 526
 527....
 528        'reset' SP <ref> LF
 529        ('from' SP <committish> LF)?
 530        LF
 531....
 532
 533For a detailed description of `<ref>` and `<committish>` see above
 534under `commit` and `from`.
 535
 536The `reset` command can also be used to create lightweight
 537(non-annotated) tags.  For example:
 538
 539====
 540        reset refs/tags/938
 541        from :938
 542====
 543
 544would create the lightweight tag `refs/tags/938` referring to
 545whatever commit mark `:938` references.
 546
 547`blob`
 548~~~~~~
 549Requests writing one file revision to the packfile.  The revision
 550is not connected to any commit; this connection must be formed in
 551a subsequent `commit` command by referencing the blob through an
 552assigned mark.
 553
 554....
 555        'blob' LF
 556        mark?
 557        data
 558....
 559
 560The mark command is optional here as some frontends have chosen
 561to generate the Git SHA-1 for the blob on their own, and feed that
 562directly to `commit`.  This is typically more work than its worth
 563however, as marks are inexpensive to store and easy to use.
 564
 565`data`
 566~~~~~~
 567Supplies raw data (for use as blob/file content, commit messages, or
 568annotated tag messages) to gfi.  Data can be supplied using an exact
 569byte count or delimited with a terminating line.  Real frontends
 570intended for production-quality conversions should always use the
 571exact byte count format, as it is more robust and performs better.
 572The delimited format is intended primarily for testing gfi.
 573
 574Exact byte count format::
 575        The frontend must specify the number of bytes of data.
 576+
 577....
 578        'data' SP <count> LF
 579        <raw> LF
 580....
 581+
 582where `<count>` is the exact number of bytes appearing within
 583`<raw>`.  The value of `<count>` is expressed as an ASCII decimal
 584integer.  The `LF` on either side of `<raw>` is not
 585included in `<count>` and will not be included in the imported data.
 586
 587Delimited format::
 588        A delimiter string is used to mark the end of the data.
 589        gfi will compute the length by searching for the delimiter.
 590        This format is primarly useful for testing and is not
 591        recommended for real data.
 592+
 593....
 594        'data' SP '<<' <delim> LF
 595        <raw> LF
 596        <delim> LF
 597....
 598+
 599where `<delim>` is the chosen delimiter string.  The string `<delim>`
 600must not appear on a line by itself within `<raw>`, as otherwise
 601gfi will think the data ends earlier than it really does.  The `LF`
 602immediately trailing `<raw>` is part of `<raw>`.  This is one of
 603the limitations of the delimited format, it is impossible to supply
 604a data chunk which does not have an LF as its last byte.
 605
 606`checkpoint`
 607~~~~~~~~~~~~
 608Forces gfi to close the current packfile and start a new one.
 609As this requires a significant amount of CPU time and disk IO
 610(to compute the overall pack SHA-1 checksum and generate the
 611corresponding index file) it can easily take several minutes for
 612a single `checkpoint` command to complete.
 613
 614....
 615        'checkpoint' LF
 616        LF
 617....
 618
 619Packfile Optimization
 620---------------------
 621When packing a blob gfi always attempts to deltify against the last
 622blob written.  Unless specifically arranged for by the frontend,
 623this will probably not be a prior version of the same file, so the
 624generated delta will not be the smallest possible.  The resulting
 625packfile will be compressed, but will not be optimal.
 626
 627Frontends which have efficient access to all revisions of a
 628single file (for example reading an RCS/CVS ,v file) can choose
 629to supply all revisions of that file as a sequence of consecutive
 630`blob` commands.  This allows gfi to deltify the different file
 631revisions against each other, saving space in the final packfile.
 632Marks can be used to later identify individual file revisions during
 633a sequence of `commit` commands.
 634
 635The packfile(s) created by gfi do not encourage good disk access
 636patterns.  This is caused by gfi writing the data in the order
 637it is received on standard input, while Git typically organizes
 638data within packfiles to make the most recent (current tip) data
 639appear before historical data.  Git also clusters commits together,
 640speeding up revision traversal through better cache locality.
 641
 642For this reason it is strongly recommended that users repack the
 643repository with `git repack -a -d` after gfi completes, allowing
 644Git to reorganize the packfiles for faster data access.  If blob
 645deltas are suboptimal (see above) then also adding the `-f` option
 646to force recomputation of all deltas can significantly reduce the
 647final packfile size (30-50% smaller can be quite typical).
 648
 649Memory Utilization
 650------------------
 651There are a number of factors which affect how much memory gfi
 652requires to perform an import.  Like critical sections of core
 653Git, gfi uses its own memory allocators to ammortize any overheads
 654associated with malloc.  In practice gfi tends to ammoritize any
 655malloc overheads to 0, due to its use of large block allocations.
 656
 657per object
 658~~~~~~~~~~
 659gfi maintains an in-memory structure for every object written in
 660this execution.  On a 32 bit system the structure is 32 bytes,
 661on a 64 bit system the structure is 40 bytes (due to the larger
 662pointer sizes).  Objects in the table are not deallocated until
 663gfi terminates.  Importing 2 million objects on a 32 bit system
 664will require approximately 64 MiB of memory.
 665
 666The object table is actually a hashtable keyed on the object name
 667(the unique SHA-1).  This storage configuration allows gfi to reuse
 668an existing or already written object and avoid writing duplicates
 669to the output packfile.  Duplicate blobs are surprisingly common
 670in an import, typically due to branch merges in the source.
 671
 672per mark
 673~~~~~~~~
 674Marks are stored in a sparse array, using 1 pointer (4 bytes or 8
 675bytes, depending on pointer size) per mark.  Although the array
 676is sparse, frontends are still strongly encouraged to use marks
 677between 1 and n, where n is the total number of marks required for
 678this import.
 679
 680per branch
 681~~~~~~~~~~
 682Branches are classified as active and inactive.  The memory usage
 683of the two classes is significantly different.
 684
 685Inactive branches are stored in a structure which uses 96 or 120
 686bytes (32 bit or 64 bit systems, respectively), plus the length of
 687the branch name (typically under 200 bytes), per branch.  gfi will
 688easily handle as many as 10,000 inactive branches in under 2 MiB
 689of memory.
 690
 691Active branches have the same overhead as inactive branches, but
 692also contain copies of every tree that has been recently modified on
 693that branch.  If subtree `include` has not been modified since the
 694branch became active, its contents will not be loaded into memory,
 695but if subtree `src` has been modified by a commit since the branch
 696became active, then its contents will be loaded in memory.
 697
 698As active branches store metadata about the files contained on that
 699branch, their in-memory storage size can grow to a considerable size
 700(see below).
 701
 702gfi automatically moves active branches to inactive status based on
 703a simple least-recently-used algorithm.  The LRU chain is updated on
 704each `commit` command.  The maximum number of active branches can be
 705increased or decreased on the command line with `--active-branches=`.
 706
 707per active tree
 708~~~~~~~~~~~~~~~
 709Trees (aka directories) use just 12 bytes of memory on top of the
 710memory required for their entries (see ``per active file'' below).
 711The cost of a tree is virtually 0, as its overhead ammortizes out
 712over the individual file entries.
 713
 714per active file entry
 715~~~~~~~~~~~~~~~~~~~~~
 716Files (and pointers to subtrees) within active trees require 52 or 64
 717bytes (32/64 bit platforms) per entry.  To conserve space, file and
 718tree names are pooled in a common string table, allowing the filename
 719``Makefile'' to use just 16 bytes (after including the string header
 720overhead) no matter how many times it occurs within the project.
 721
 722The active branch LRU, when coupled with the filename string pool
 723and lazy loading of subtrees, allows gfi to efficiently import
 724projects with 2,000+ branches and 45,114+ files in a very limited
 725memory footprint (less than 2.7 MiB per active branch).
 726
 727
 728Author
 729------
 730Written by Shawn O. Pearce <spearce@spearce.org>.
 731
 732Documentation
 733--------------
 734Documentation by Shawn O. Pearce <spearce@spearce.org>.
 735
 736GIT
 737---
 738Part of the gitlink:git[7] suite
 739