1git-fast-import(1) 2================== 3 4NAME 5---- 6git-fast-import - Backend for fast Git data importers. 7 8 9SYNOPSIS 10-------- 11frontend | 'git-fast-import' [options] 12 13DESCRIPTION 14----------- 15This program is usually not what the end user wants to run directly. 16Most end users want to use one of the existing frontend programs, 17which parses a specific type of foreign source and feeds the contents 18stored there to git-fast-import (gfi). 19 20gfi reads a mixed command/data stream from standard input and 21writes one or more packfiles directly into the current repository. 22When EOF is received on standard input, fast import writes out 23updated branch and tag refs, fully updating the current repository 24with the newly imported data. 25 26The gfi backend itself can import into an empty repository (one that 27has already been initialized by gitlink:git-init[1]) or incrementally 28update an existing populated repository. Whether or not incremental 29imports are supported from a particular foreign source depends on 30the frontend program in use. 31 32 33OPTIONS 34------- 35--date-format=<fmt>:: 36 Specify the type of dates the frontend will supply to 37 gfi within `author`, `committer` and `tagger` commands. 38 See ``Date Formats'' below for details about which formats 39 are supported, and their syntax. 40 41--force:: 42 Force updating modified existing branches, even if doing 43 so would cause commits to be lost (as the new commit does 44 not contain the old commit). 45 46--max-pack-size=<n>:: 47 Maximum size of each output packfile, expressed in MiB. 48 The default is 4096 (4 GiB) as that is the maximum allowed 49 packfile size (due to file format limitations). Some 50 importers may wish to lower this, such as to ensure the 51 resulting packfiles fit on CDs. 52 53--depth=<n>:: 54 Maximum delta depth, for blob and tree deltification. 55 Default is 10. 56 57--active-branches=<n>:: 58 Maximum number of branches to maintain active at once. 59 See ``Memory Utilization'' below for details. Default is 5. 60 61--export-marks=<file>:: 62 Dumps the internal marks table to <file> when complete. 63 Marks are written one per line as `:markid SHA-1`. 64 Frontends can use this file to validate imports after they 65 have been completed. 66 67Performance 68----------- 69The design of gfi allows it to import large projects in a minimum 70amount of memory usage and processing time. Assuming the frontend 71is able to keep up with gfi and feed it a constant stream of data, 72import times for projects holding 10+ years of history and containing 73100,000+ individual commits are generally completed in just 1-2 74hours on quite modest (~$2,000 USD) hardware. 75 76Most bottlenecks appear to be in foreign source data access (the 77source just cannot extract revisions fast enough) or disk IO (gfi 78writes as fast as the disk will take the data). Imports will run 79faster if the source data is stored on a different drive than the 80destination Git repository (due to less IO contention). 81 82 83Development Cost 84---------------- 85A typical frontend for gfi tends to weigh in at approximately 200 86lines of Perl/Python/Ruby code. Most developers have been able to 87create working importers in just a couple of hours, even though it 88is their first exposure to gfi, and sometimes even to Git. This is 89an ideal situation, given that most conversion tools are throw-away 90(use once, and never look back). 91 92 93Parallel Operation 94------------------ 95Like `git-push` or `git-fetch`, imports handled by gfi are safe to 96run alongside parallel `git repack -a -d` or `git gc` invocations, 97or any other Git operation (including `git prune`, as loose objects 98are never used by gfi). 99 100gfi does not lock the branch or tag refs it is actively importing. 101After the import, during its ref update phase, gfi tests each 102existing branch ref to verify the update will be a fast-forward 103update (the commit stored in the ref is contained in the new 104history of the commit to be written). If the update is not a 105fast-forward update, gfi will skip updating that ref and instead 106prints a warning message. gfi will always attempt to update all 107branch refs, and does not stop on the first failure. 108 109Branch updates can be forced with `--force`, but its recommended that 110this only be used on an otherwise quiet repository. Using `--force` 111is not necessary for an initial import into an empty repository. 112 113 114Technical Discussion 115-------------------- 116gfi tracks a set of branches in memory. Any branch can be created 117or modified at any point during the import process by sending a 118`commit` command on the input stream. This design allows a frontend 119program to process an unlimited number of branches simultaneously, 120generating commits in the order they are available from the source 121data. It also simplifies the frontend programs considerably. 122 123gfi does not use or alter the current working directory, or any 124file within it. (It does however update the current Git repository, 125as referenced by `GIT_DIR`.) Therefore an import frontend may use 126the working directory for its own purposes, such as extracting file 127revisions from the foreign source. This ignorance of the working 128directory also allows gfi to run very quickly, as it does not 129need to perform any costly file update operations when switching 130between branches. 131 132Input Format 133------------ 134With the exception of raw file data (which Git does not interpret) 135the gfi input format is text (ASCII) based. This text based 136format simplifies development and debugging of frontend programs, 137especially when a higher level language such as Perl, Python or 138Ruby is being used. 139 140gfi is very strict about its input. Where we say SP below we mean 141*exactly* one space. Likewise LF means one (and only one) linefeed. 142Supplying additional whitespace characters will cause unexpected 143results, such as branch names or file names with leading or trailing 144spaces in their name, or early termination of gfi when it encounters 145unexpected input. 146 147Date Formats 148~~~~~~~~~~~~ 149The following date formats are supported. A frontend should select 150the format it will use for this import by passing the format name 151in the `--date-format=<fmt>` command line option. 152 153`raw`:: 154 This is the Git native format and is `<time> SP <offutc>`. 155 It is also gfi's default format, if `--date-format` was 156 not specified. 157+ 158The time of the event is specified by `<time>` as the number of 159seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is 160written as an ASCII decimal integer. 161+ 162The local offset is specified by `<offutc>` as a positive or negative 163offset from UTC. For example EST (which is 5 hours behind UTC) 164would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''. 165The local offset does not affect `<time>`; it is used only as an 166advisement to help formatting routines display the timestamp. 167+ 168If the local offset is not available in the source material, use 169``+0000'', or the most common local offset. For example many 170organizations have a CVS repository which has only ever been accessed 171by users who are located in the same location and timezone. In this 172case the offset from UTC can be easily assumed. 173+ 174Unlike the `rfc2822` format, this format is very strict. Any 175variation in formatting will cause gfi to reject the value. 176 177`rfc2822`:: 178 This is the standard email format as described by RFC 2822. 179+ 180An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git 181parser is accurate, but a little on the lenient side. Its the 182same parser used by gitlink:git-am[1] when applying patches 183received from email. 184+ 185Some malformed strings may be accepted as valid dates. In some of 186these cases Git will still be able to obtain the correct date from 187the malformed string. There are also some types of malformed 188strings which Git will parse wrong, and yet consider valid. 189Seriously malformed strings will be rejected. 190+ 191Unlike the `raw` format above, the timezone/UTC offset information 192contained in an RFC 2822 date string is used to adjust the date 193value to UTC prior to storage. Therefore it is important that 194this information be as accurate as possible. 195+ 196If the source material is formatted in RFC 2822 style dates, 197the frontend should let gfi handle the parsing and conversion 198(rather than attempting to do it itself) as the Git parser has 199been well tested in the wild. 200+ 201Frontends should prefer the `raw` format if the source material 202is already in UNIX-epoch format, or is easily convertible to 203that format, as there is no ambiguity in parsing. 204 205`now`:: 206 Always use the current time and timezone. The literal 207 `now` must always be supplied for `<when>`. 208+ 209This is a toy format. The current time and timezone of this system 210is always copied into the identity string at the time it is being 211created by gfi. There is no way to specify a different time or 212timezone. 213+ 214This particular format is supplied as its short to implement and 215may be useful to a process that wants to create a new commit 216right now, without needing to use a working directory or 217gitlink:git-update-index[1]. 218+ 219If separate `author` and `committer` commands are used in a `commit` 220the timestamps may not match, as the system clock will be polled 221twice (once for each command). The only way to ensure that both 222author and committer identity information has the same timestamp 223is to omit `author` (thus copying from `committer`) or to use a 224date format other than `now`. 225 226Commands 227~~~~~~~~ 228gfi accepts several commands to update the current repository 229and control the current import process. More detailed discussion 230(with examples) of each command follows later. 231 232`commit`:: 233 Creates a new branch or updates an existing branch by 234 creating a new commit and updating the branch to point at 235 the newly created commit. 236 237`tag`:: 238 Creates an annotated tag object from an existing commit or 239 branch. Lightweight tags are not supported by this command, 240 as they are not recommended for recording meaningful points 241 in time. 242 243`reset`:: 244 Reset an existing branch (or a new branch) to a specific 245 revision. This command must be used to change a branch to 246 a specific revision without making a commit on it. 247 248`blob`:: 249 Convert raw file data into a blob, for future use in a 250 `commit` command. This command is optional and is not 251 needed to perform an import. 252 253`checkpoint`:: 254 Forces gfi to close the current packfile, generate its 255 unique SHA-1 checksum and index, and start a new packfile. 256 This command is optional and is not needed to perform 257 an import. 258 259`commit` 260~~~~~~~~ 261Create or update a branch with a new commit, recording one logical 262change to the project. 263 264.... 265 'commit' SP <ref> LF 266 mark? 267 ('author' SP <name> SP LT <email> GT SP <when> LF)? 268 'committer' SP <name> SP LT <email> GT SP <when> LF 269 data 270 ('from' SP <committish> LF)? 271 ('merge' SP <committish> LF)? 272 (filemodify | filedelete | filedeleteall)* 273 LF 274.... 275 276where `<ref>` is the name of the branch to make the commit on. 277Typically branch names are prefixed with `refs/heads/` in 278Git, so importing the CVS branch symbol `RELENG-1_0` would use 279`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of 280`<ref>` must be a valid refname in Git. As `LF` is not valid in 281a Git refname, no quoting or escaping syntax is supported here. 282 283A `mark` command may optionally appear, requesting gfi to save a 284reference to the newly created commit for future use by the frontend 285(see below for format). It is very common for frontends to mark 286every commit they create, thereby allowing future branch creation 287from any imported commit. 288 289The `data` command following `committer` must supply the commit 290message (see below for `data` command syntax). To import an empty 291commit message use a 0 length data. Commit messages are free-form 292and are not interpreted by Git. Currently they must be encoded in 293UTF-8, as gfi does not permit other encodings to be specified. 294 295Zero or more `filemodify`, `filedelete` and `filedeleteall` commands 296may be included to update the contents of the branch prior to 297creating the commit. These commands may be supplied in any order. 298However it is recommended that a `filedeleteall` command preceed 299all `filemodify` commands in the same commit, as `filedeleteall` 300wipes the branch clean (see below). 301 302`author` 303^^^^^^^^ 304An `author` command may optionally appear, if the author information 305might differ from the committer information. If `author` is omitted 306then gfi will automatically use the committer's information for 307the author portion of the commit. See below for a description of 308the fields in `author`, as they are identical to `committer`. 309 310`committer` 311^^^^^^^^^^^ 312The `committer` command indicates who made this commit, and when 313they made it. 314 315Here `<name>` is the person's display name (for example 316``Com M Itter'') and `<email>` is the person's email address 317(``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c) 318and greater-than (\x3e) symbols. These are required to delimit 319the email address from the other fields in the line. Note that 320`<name>` is free-form and may contain any sequence of bytes, except 321`LT` and `LF`. It is typically UTF-8 encoded. 322 323The time of the change is specified by `<when>` using the date format 324that was selected by the `--date-format=<fmt>` command line option. 325See ``Date Formats'' above for the set of supported formats, and 326their syntax. 327 328`from` 329^^^^^^ 330Only valid for the first commit made on this branch by this 331gfi process. The `from` command is used to specify the commit 332to initialize this branch from. This revision will be the first 333ancestor of the new commit. 334 335Omitting the `from` command in the first commit of a new branch will 336cause gfi to create that commit with no ancestor. This tends to be 337desired only for the initial commit of a project. Omitting the 338`from` command on existing branches is required, as the current 339commit on that branch is automatically assumed to be the first 340ancestor of the new commit. 341 342As `LF` is not valid in a Git refname or SHA-1 expression, no 343quoting or escaping syntax is supported within `<committish>`. 344 345Here `<committish>` is any of the following: 346 347* The name of an existing branch already in gfi's internal branch 348 table. If gfi doesn't know the name, its treated as a SHA-1 349 expression. 350 351* A mark reference, `:<idnum>`, where `<idnum>` is the mark number. 352+ 353The reason gfi uses `:` to denote a mark reference is this character 354is not legal in a Git branch name. The leading `:` makes it easy 355to distingush between the mark 42 (`:42`) and the branch 42 (`42` 356or `refs/heads/42`), or an abbreviated SHA-1 which happened to 357consist only of base-10 digits. 358+ 359Marks must be declared (via `mark`) before they can be used. 360 361* A complete 40 byte or abbreviated commit SHA-1 in hex. 362 363* Any valid Git SHA-1 expression that resolves to a commit. See 364 ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details. 365 366The special case of restarting an incremental import from the 367current branch value should be written as: 368---- 369 from refs/heads/branch^0 370---- 371The `^0` suffix is necessary as gfi does not permit a branch to 372start from itself, and the branch is created in memory before the 373`from` command is even read from the input. Adding `^0` will force 374gfi to resolve the commit through Git's revision parsing library, 375rather than its internal branch table, thereby loading in the 376existing value of the branch. 377 378`merge` 379^^^^^^^ 380Includes one additional ancestor commit, and makes the current 381commit a merge commit. An unlimited number of `merge` commands per 382commit are permitted by gfi, thereby establishing an n-way merge. 383However Git's other tools never create commits with more than 15 384additional ancestors (forming a 16-way merge). For this reason 385it is suggested that frontends do not use more than 15 `merge` 386commands per commit. 387 388Here `<committish>` is any of the commit specification expressions 389also accepted by `from` (see above). 390 391`filemodify` 392^^^^^^^^^^^^ 393Included in a `commit` command to add a new file or change the 394content of an existing file. This command has two different means 395of specifying the content of the file. 396 397External data format:: 398 The data content for the file was already supplied by a prior 399 `blob` command. The frontend just needs to connect it. 400+ 401.... 402 'M' SP <mode> SP <dataref> SP <path> LF 403.... 404+ 405Here `<dataref>` can be either a mark reference (`:<idnum>`) 406set by a prior `blob` command, or a full 40-byte SHA-1 of an 407existing Git blob object. 408 409Inline data format:: 410 The data content for the file has not been supplied yet. 411 The frontend wants to supply it as part of this modify 412 command. 413+ 414.... 415 'M' SP <mode> SP 'inline' SP <path> LF 416 data 417.... 418+ 419See below for a detailed description of the `data` command. 420 421In both formats `<mode>` is the type of file entry, specified 422in octal. Git only supports the following modes: 423 424* `100644` or `644`: A normal (not-executable) file. The majority 425 of files in most projects use this mode. If in doubt, this is 426 what you want. 427* `100755` or `755`: A normal, but executable, file. 428* `120000`: A symlink, the content of the file will be the link target. 429 430In both formats `<path>` is the complete path of the file to be added 431(if not already existing) or modified (if already existing). 432 433A `<path>` string must use UNIX-style directory seperators (forward 434slash `/`), may contain any byte other than `LF`, and must not 435start with double quote (`"`). 436 437If an `LF` or double quote must be encoded into `<path>` shell-style 438quoting should be used, e.g. `"path/with\n and \" in it"`. 439 440The value of `<path>` must be in canoncial form. That is it must not: 441 442* contain an empty directory component (e.g. `foo//bar` is invalid), 443* end with a directory seperator (e.g. `foo/` is invalid), 444* start with a directory seperator (e.g. `/foo` is invalid), 445* contain the special component `.` or `..` (e.g. `foo/./bar` and 446 `foo/../bar` are invalid). 447 448It is recommended that `<path>` always be encoded using UTF-8. 449 450`filedelete` 451^^^^^^^^^^^^ 452Included in a `commit` command to remove a file from the branch. 453If the file removal makes its directory empty, the directory will 454be automatically removed too. This cascades up the tree until the 455first non-empty directory or the root is reached. 456 457.... 458 'D' SP <path> LF 459.... 460 461here `<path>` is the complete path of the file to be removed. 462See `filemodify` above for a detailed description of `<path>`. 463 464`filedeleteall` 465^^^^^^^^^^^^^^^ 466Included in a `commit` command to remove all files (and also all 467directories) from the branch. This command resets the internal 468branch structure to have no files in it, allowing the frontend 469to subsequently add all interesting files from scratch. 470 471.... 472 'deleteall' LF 473.... 474 475This command is extremely useful if the frontend does not know 476(or does not care to know) what files are currently on the branch, 477and therefore cannot generate the proper `filedelete` commands to 478update the content. 479 480Issuing a `filedeleteall` followed by the needed `filemodify` 481commands to set the correct content will produce the same results 482as sending only the needed `filemodify` and `filedelete` commands. 483The `filedeleteall` approach may however require gfi to use slightly 484more memory per active branch (less than 1 MiB for even most large 485projects); so frontends that can easily obtain only the affected 486paths for a commit are encouraged to do so. 487 488`mark` 489~~~~~~ 490Arranges for gfi to save a reference to the current object, allowing 491the frontend to recall this object at a future point in time, without 492knowing its SHA-1. Here the current object is the object creation 493command the `mark` command appears within. This can be `commit`, 494`tag`, and `blob`, but `commit` is the most common usage. 495 496.... 497 'mark' SP ':' <idnum> LF 498.... 499 500where `<idnum>` is the number assigned by the frontend to this mark. 501The value of `<idnum>` is expressed as an ASCII decimal integer. 502The value 0 is reserved and cannot be used as 503a mark. Only values greater than or equal to 1 may be used as marks. 504 505New marks are created automatically. Existing marks can be moved 506to another object simply by reusing the same `<idnum>` in another 507`mark` command. 508 509`tag` 510~~~~~ 511Creates an annotated tag referring to a specific commit. To create 512lightweight (non-annotated) tags see the `reset` command below. 513 514.... 515 'tag' SP <name> LF 516 'from' SP <committish> LF 517 'tagger' SP <name> SP LT <email> GT SP <when> LF 518 data 519 LF 520.... 521 522where `<name>` is the name of the tag to create. 523 524Tag names are automatically prefixed with `refs/tags/` when stored 525in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would 526use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the 527corresponding ref as `refs/tags/RELENG-1_0-FINAL`. 528 529The value of `<name>` must be a valid refname in Git and therefore 530may contain forward slashes. As `LF` is not valid in a Git refname, 531no quoting or escaping syntax is supported here. 532 533The `from` command is the same as in the `commit` command; see 534above for details. 535 536The `tagger` command uses the same format as `committer` within 537`commit`; again see above for details. 538 539The `data` command following `tagger` must supply the annotated tag 540message (see below for `data` command syntax). To import an empty 541tag message use a 0 length data. Tag messages are free-form and are 542not interpreted by Git. Currently they must be encoded in UTF-8, 543as gfi does not permit other encodings to be specified. 544 545Signing annotated tags during import from within gfi is not 546supported. Trying to include your own PGP/GPG signature is not 547recommended, as the frontend does not (easily) have access to the 548complete set of bytes which normally goes into such a signature. 549If signing is required, create lightweight tags from within gfi with 550`reset`, then create the annotated versions of those tags offline 551with the standard gitlink:git-tag[1] process. 552 553`reset` 554~~~~~~~ 555Creates (or recreates) the named branch, optionally starting from 556a specific revision. The reset command allows a frontend to issue 557a new `from` command for an existing branch, or to create a new 558branch from an existing commit without creating a new commit. 559 560.... 561 'reset' SP <ref> LF 562 ('from' SP <committish> LF)? 563 LF 564.... 565 566For a detailed description of `<ref>` and `<committish>` see above 567under `commit` and `from`. 568 569The `reset` command can also be used to create lightweight 570(non-annotated) tags. For example: 571 572==== 573 reset refs/tags/938 574 from :938 575==== 576 577would create the lightweight tag `refs/tags/938` referring to 578whatever commit mark `:938` references. 579 580`blob` 581~~~~~~ 582Requests writing one file revision to the packfile. The revision 583is not connected to any commit; this connection must be formed in 584a subsequent `commit` command by referencing the blob through an 585assigned mark. 586 587.... 588 'blob' LF 589 mark? 590 data 591.... 592 593The mark command is optional here as some frontends have chosen 594to generate the Git SHA-1 for the blob on their own, and feed that 595directly to `commit`. This is typically more work than its worth 596however, as marks are inexpensive to store and easy to use. 597 598`data` 599~~~~~~ 600Supplies raw data (for use as blob/file content, commit messages, or 601annotated tag messages) to gfi. Data can be supplied using an exact 602byte count or delimited with a terminating line. Real frontends 603intended for production-quality conversions should always use the 604exact byte count format, as it is more robust and performs better. 605The delimited format is intended primarily for testing gfi. 606 607Exact byte count format:: 608 The frontend must specify the number of bytes of data. 609+ 610.... 611 'data' SP <count> LF 612 <raw> LF 613.... 614+ 615where `<count>` is the exact number of bytes appearing within 616`<raw>`. The value of `<count>` is expressed as an ASCII decimal 617integer. The `LF` on either side of `<raw>` is not 618included in `<count>` and will not be included in the imported data. 619 620Delimited format:: 621 A delimiter string is used to mark the end of the data. 622 gfi will compute the length by searching for the delimiter. 623 This format is primarly useful for testing and is not 624 recommended for real data. 625+ 626.... 627 'data' SP '<<' <delim> LF 628 <raw> LF 629 <delim> LF 630.... 631+ 632where `<delim>` is the chosen delimiter string. The string `<delim>` 633must not appear on a line by itself within `<raw>`, as otherwise 634gfi will think the data ends earlier than it really does. The `LF` 635immediately trailing `<raw>` is part of `<raw>`. This is one of 636the limitations of the delimited format, it is impossible to supply 637a data chunk which does not have an LF as its last byte. 638 639`checkpoint` 640~~~~~~~~~~~~ 641Forces gfi to close the current packfile and start a new one. 642As this requires a significant amount of CPU time and disk IO 643(to compute the overall pack SHA-1 checksum and generate the 644corresponding index file) it can easily take several minutes for 645a single `checkpoint` command to complete. 646 647.... 648 'checkpoint' LF 649 LF 650.... 651 652Packfile Optimization 653--------------------- 654When packing a blob gfi always attempts to deltify against the last 655blob written. Unless specifically arranged for by the frontend, 656this will probably not be a prior version of the same file, so the 657generated delta will not be the smallest possible. The resulting 658packfile will be compressed, but will not be optimal. 659 660Frontends which have efficient access to all revisions of a 661single file (for example reading an RCS/CVS ,v file) can choose 662to supply all revisions of that file as a sequence of consecutive 663`blob` commands. This allows gfi to deltify the different file 664revisions against each other, saving space in the final packfile. 665Marks can be used to later identify individual file revisions during 666a sequence of `commit` commands. 667 668The packfile(s) created by gfi do not encourage good disk access 669patterns. This is caused by gfi writing the data in the order 670it is received on standard input, while Git typically organizes 671data within packfiles to make the most recent (current tip) data 672appear before historical data. Git also clusters commits together, 673speeding up revision traversal through better cache locality. 674 675For this reason it is strongly recommended that users repack the 676repository with `git repack -a -d` after gfi completes, allowing 677Git to reorganize the packfiles for faster data access. If blob 678deltas are suboptimal (see above) then also adding the `-f` option 679to force recomputation of all deltas can significantly reduce the 680final packfile size (30-50% smaller can be quite typical). 681 682Memory Utilization 683------------------ 684There are a number of factors which affect how much memory gfi 685requires to perform an import. Like critical sections of core 686Git, gfi uses its own memory allocators to ammortize any overheads 687associated with malloc. In practice gfi tends to ammoritize any 688malloc overheads to 0, due to its use of large block allocations. 689 690per object 691~~~~~~~~~~ 692gfi maintains an in-memory structure for every object written in 693this execution. On a 32 bit system the structure is 32 bytes, 694on a 64 bit system the structure is 40 bytes (due to the larger 695pointer sizes). Objects in the table are not deallocated until 696gfi terminates. Importing 2 million objects on a 32 bit system 697will require approximately 64 MiB of memory. 698 699The object table is actually a hashtable keyed on the object name 700(the unique SHA-1). This storage configuration allows gfi to reuse 701an existing or already written object and avoid writing duplicates 702to the output packfile. Duplicate blobs are surprisingly common 703in an import, typically due to branch merges in the source. 704 705per mark 706~~~~~~~~ 707Marks are stored in a sparse array, using 1 pointer (4 bytes or 8 708bytes, depending on pointer size) per mark. Although the array 709is sparse, frontends are still strongly encouraged to use marks 710between 1 and n, where n is the total number of marks required for 711this import. 712 713per branch 714~~~~~~~~~~ 715Branches are classified as active and inactive. The memory usage 716of the two classes is significantly different. 717 718Inactive branches are stored in a structure which uses 96 or 120 719bytes (32 bit or 64 bit systems, respectively), plus the length of 720the branch name (typically under 200 bytes), per branch. gfi will 721easily handle as many as 10,000 inactive branches in under 2 MiB 722of memory. 723 724Active branches have the same overhead as inactive branches, but 725also contain copies of every tree that has been recently modified on 726that branch. If subtree `include` has not been modified since the 727branch became active, its contents will not be loaded into memory, 728but if subtree `src` has been modified by a commit since the branch 729became active, then its contents will be loaded in memory. 730 731As active branches store metadata about the files contained on that 732branch, their in-memory storage size can grow to a considerable size 733(see below). 734 735gfi automatically moves active branches to inactive status based on 736a simple least-recently-used algorithm. The LRU chain is updated on 737each `commit` command. The maximum number of active branches can be 738increased or decreased on the command line with `--active-branches=`. 739 740per active tree 741~~~~~~~~~~~~~~~ 742Trees (aka directories) use just 12 bytes of memory on top of the 743memory required for their entries (see ``per active file'' below). 744The cost of a tree is virtually 0, as its overhead ammortizes out 745over the individual file entries. 746 747per active file entry 748~~~~~~~~~~~~~~~~~~~~~ 749Files (and pointers to subtrees) within active trees require 52 or 64 750bytes (32/64 bit platforms) per entry. To conserve space, file and 751tree names are pooled in a common string table, allowing the filename 752``Makefile'' to use just 16 bytes (after including the string header 753overhead) no matter how many times it occurs within the project. 754 755The active branch LRU, when coupled with the filename string pool 756and lazy loading of subtrees, allows gfi to efficiently import 757projects with 2,000+ branches and 45,114+ files in a very limited 758memory footprint (less than 2.7 MiB per active branch). 759 760 761Author 762------ 763Written by Shawn O. Pearce <spearce@spearce.org>. 764 765Documentation 766-------------- 767Documentation by Shawn O. Pearce <spearce@spearce.org>. 768 769GIT 770--- 771Part of the gitlink:git[7] suite 772