1git-fast-import(1) 2================== 3 4NAME 5---- 6git-fast-import - Backend for fast Git data importers 7 8 9SYNOPSIS 10-------- 11frontend | 'git-fast-import' [options] 12 13DESCRIPTION 14----------- 15This program is usually not what the end user wants to run directly. 16Most end users want to use one of the existing frontend programs, 17which parses a specific type of foreign source and feeds the contents 18stored there to git-fast-import. 19 20fast-import reads a mixed command/data stream from standard input and 21writes one or more packfiles directly into the current repository. 22When EOF is received on standard input, fast import writes out 23updated branch and tag refs, fully updating the current repository 24with the newly imported data. 25 26The fast-import backend itself can import into an empty repository (one that 27has already been initialized by gitlink:git-init[1]) or incrementally 28update an existing populated repository. Whether or not incremental 29imports are supported from a particular foreign source depends on 30the frontend program in use. 31 32 33OPTIONS 34------- 35--date-format=<fmt>:: 36 Specify the type of dates the frontend will supply to 37 fast-import within `author`, `committer` and `tagger` commands. 38 See ``Date Formats'' below for details about which formats 39 are supported, and their syntax. 40 41--force:: 42 Force updating modified existing branches, even if doing 43 so would cause commits to be lost (as the new commit does 44 not contain the old commit). 45 46--max-pack-size=<n>:: 47 Maximum size of each output packfile, expressed in MiB. 48 The default is 4096 (4 GiB) as that is the maximum allowed 49 packfile size (due to file format limitations). Some 50 importers may wish to lower this, such as to ensure the 51 resulting packfiles fit on CDs. 52 53--depth=<n>:: 54 Maximum delta depth, for blob and tree deltification. 55 Default is 10. 56 57--active-branches=<n>:: 58 Maximum number of branches to maintain active at once. 59 See ``Memory Utilization'' below for details. Default is 5. 60 61--export-marks=<file>:: 62 Dumps the internal marks table to <file> when complete. 63 Marks are written one per line as `:markid SHA-1`. 64 Frontends can use this file to validate imports after they 65 have been completed, or to save the marks table across 66 incremental runs. As <file> is only opened and truncated 67 at checkpoint (or completion) the same path can also be 68 safely given to \--import-marks. 69 70--import-marks=<file>:: 71 Before processing any input, load the marks specified in 72 <file>. The input file must exist, must be readable, and 73 must use the same format as produced by \--export-marks. 74 Multiple options may be supplied to import more than one 75 set of marks. If a mark is defined to different values, 76 the last file wins. 77 78--export-pack-edges=<file>:: 79 After creating a packfile, print a line of data to 80 <file> listing the filename of the packfile and the last 81 commit on each branch that was written to that packfile. 82 This information may be useful after importing projects 83 whose total object set exceeds the 4 GiB packfile limit, 84 as these commits can be used as edge points during calls 85 to gitlink:git-pack-objects[1]. 86 87--quiet:: 88 Disable all non-fatal output, making fast-import silent when it 89 is successful. This option disables the output shown by 90 \--stats. 91 92--stats:: 93 Display some basic statistics about the objects fast-import has 94 created, the packfiles they were stored into, and the 95 memory used by fast-import during this run. Showing this output 96 is currently the default, but can be disabled with \--quiet. 97 98 99Performance 100----------- 101The design of fast-import allows it to import large projects in a minimum 102amount of memory usage and processing time. Assuming the frontend 103is able to keep up with fast-import and feed it a constant stream of data, 104import times for projects holding 10+ years of history and containing 105100,000+ individual commits are generally completed in just 1-2 106hours on quite modest (~$2,000 USD) hardware. 107 108Most bottlenecks appear to be in foreign source data access (the 109source just cannot extract revisions fast enough) or disk IO (fast-import 110writes as fast as the disk will take the data). Imports will run 111faster if the source data is stored on a different drive than the 112destination Git repository (due to less IO contention). 113 114 115Development Cost 116---------------- 117A typical frontend for fast-import tends to weigh in at approximately 200 118lines of Perl/Python/Ruby code. Most developers have been able to 119create working importers in just a couple of hours, even though it 120is their first exposure to fast-import, and sometimes even to Git. This is 121an ideal situation, given that most conversion tools are throw-away 122(use once, and never look back). 123 124 125Parallel Operation 126------------------ 127Like `git-push` or `git-fetch`, imports handled by fast-import are safe to 128run alongside parallel `git repack -a -d` or `git gc` invocations, 129or any other Git operation (including `git prune`, as loose objects 130are never used by fast-import). 131 132fast-import does not lock the branch or tag refs it is actively importing. 133After the import, during its ref update phase, fast-import tests each 134existing branch ref to verify the update will be a fast-forward 135update (the commit stored in the ref is contained in the new 136history of the commit to be written). If the update is not a 137fast-forward update, fast-import will skip updating that ref and instead 138prints a warning message. fast-import will always attempt to update all 139branch refs, and does not stop on the first failure. 140 141Branch updates can be forced with \--force, but its recommended that 142this only be used on an otherwise quiet repository. Using \--force 143is not necessary for an initial import into an empty repository. 144 145 146Technical Discussion 147-------------------- 148fast-import tracks a set of branches in memory. Any branch can be created 149or modified at any point during the import process by sending a 150`commit` command on the input stream. This design allows a frontend 151program to process an unlimited number of branches simultaneously, 152generating commits in the order they are available from the source 153data. It also simplifies the frontend programs considerably. 154 155fast-import does not use or alter the current working directory, or any 156file within it. (It does however update the current Git repository, 157as referenced by `GIT_DIR`.) Therefore an import frontend may use 158the working directory for its own purposes, such as extracting file 159revisions from the foreign source. This ignorance of the working 160directory also allows fast-import to run very quickly, as it does not 161need to perform any costly file update operations when switching 162between branches. 163 164Input Format 165------------ 166With the exception of raw file data (which Git does not interpret) 167the fast-import input format is text (ASCII) based. This text based 168format simplifies development and debugging of frontend programs, 169especially when a higher level language such as Perl, Python or 170Ruby is being used. 171 172fast-import is very strict about its input. Where we say SP below we mean 173*exactly* one space. Likewise LF means one (and only one) linefeed. 174Supplying additional whitespace characters will cause unexpected 175results, such as branch names or file names with leading or trailing 176spaces in their name, or early termination of fast-import when it encounters 177unexpected input. 178 179Date Formats 180~~~~~~~~~~~~ 181The following date formats are supported. A frontend should select 182the format it will use for this import by passing the format name 183in the \--date-format=<fmt> command line option. 184 185`raw`:: 186 This is the Git native format and is `<time> SP <offutc>`. 187 It is also fast-import's default format, if \--date-format was 188 not specified. 189+ 190The time of the event is specified by `<time>` as the number of 191seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is 192written as an ASCII decimal integer. 193+ 194The local offset is specified by `<offutc>` as a positive or negative 195offset from UTC. For example EST (which is 5 hours behind UTC) 196would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''. 197The local offset does not affect `<time>`; it is used only as an 198advisement to help formatting routines display the timestamp. 199+ 200If the local offset is not available in the source material, use 201``+0000'', or the most common local offset. For example many 202organizations have a CVS repository which has only ever been accessed 203by users who are located in the same location and timezone. In this 204case a reasonable offset from UTC could be assumed. 205+ 206Unlike the `rfc2822` format, this format is very strict. Any 207variation in formatting will cause fast-import to reject the value. 208 209`rfc2822`:: 210 This is the standard email format as described by RFC 2822. 211+ 212An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git 213parser is accurate, but a little on the lenient side. It is the 214same parser used by gitlink:git-am[1] when applying patches 215received from email. 216+ 217Some malformed strings may be accepted as valid dates. In some of 218these cases Git will still be able to obtain the correct date from 219the malformed string. There are also some types of malformed 220strings which Git will parse wrong, and yet consider valid. 221Seriously malformed strings will be rejected. 222+ 223Unlike the `raw` format above, the timezone/UTC offset information 224contained in an RFC 2822 date string is used to adjust the date 225value to UTC prior to storage. Therefore it is important that 226this information be as accurate as possible. 227+ 228If the source material uses RFC 2822 style dates, 229the frontend should let fast-import handle the parsing and conversion 230(rather than attempting to do it itself) as the Git parser has 231been well tested in the wild. 232+ 233Frontends should prefer the `raw` format if the source material 234already uses UNIX-epoch format, can be coaxed to give dates in that 235format, or its format is easiliy convertible to it, as there is no 236ambiguity in parsing. 237 238`now`:: 239 Always use the current time and timezone. The literal 240 `now` must always be supplied for `<when>`. 241+ 242This is a toy format. The current time and timezone of this system 243is always copied into the identity string at the time it is being 244created by fast-import. There is no way to specify a different time or 245timezone. 246+ 247This particular format is supplied as its short to implement and 248may be useful to a process that wants to create a new commit 249right now, without needing to use a working directory or 250gitlink:git-update-index[1]. 251+ 252If separate `author` and `committer` commands are used in a `commit` 253the timestamps may not match, as the system clock will be polled 254twice (once for each command). The only way to ensure that both 255author and committer identity information has the same timestamp 256is to omit `author` (thus copying from `committer`) or to use a 257date format other than `now`. 258 259Commands 260~~~~~~~~ 261fast-import accepts several commands to update the current repository 262and control the current import process. More detailed discussion 263(with examples) of each command follows later. 264 265`commit`:: 266 Creates a new branch or updates an existing branch by 267 creating a new commit and updating the branch to point at 268 the newly created commit. 269 270`tag`:: 271 Creates an annotated tag object from an existing commit or 272 branch. Lightweight tags are not supported by this command, 273 as they are not recommended for recording meaningful points 274 in time. 275 276`reset`:: 277 Reset an existing branch (or a new branch) to a specific 278 revision. This command must be used to change a branch to 279 a specific revision without making a commit on it. 280 281`blob`:: 282 Convert raw file data into a blob, for future use in a 283 `commit` command. This command is optional and is not 284 needed to perform an import. 285 286`checkpoint`:: 287 Forces fast-import to close the current packfile, generate its 288 unique SHA-1 checksum and index, and start a new packfile. 289 This command is optional and is not needed to perform 290 an import. 291 292`commit` 293~~~~~~~~ 294Create or update a branch with a new commit, recording one logical 295change to the project. 296 297.... 298 'commit' SP <ref> LF 299 mark? 300 ('author' SP <name> SP LT <email> GT SP <when> LF)? 301 'committer' SP <name> SP LT <email> GT SP <when> LF 302 data 303 ('from' SP <committish> LF)? 304 ('merge' SP <committish> LF)? 305 (filemodify | filedelete | filedeleteall)* 306 LF 307.... 308 309where `<ref>` is the name of the branch to make the commit on. 310Typically branch names are prefixed with `refs/heads/` in 311Git, so importing the CVS branch symbol `RELENG-1_0` would use 312`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of 313`<ref>` must be a valid refname in Git. As `LF` is not valid in 314a Git refname, no quoting or escaping syntax is supported here. 315 316A `mark` command may optionally appear, requesting fast-import to save a 317reference to the newly created commit for future use by the frontend 318(see below for format). It is very common for frontends to mark 319every commit they create, thereby allowing future branch creation 320from any imported commit. 321 322The `data` command following `committer` must supply the commit 323message (see below for `data` command syntax). To import an empty 324commit message use a 0 length data. Commit messages are free-form 325and are not interpreted by Git. Currently they must be encoded in 326UTF-8, as fast-import does not permit other encodings to be specified. 327 328Zero or more `filemodify`, `filedelete` and `filedeleteall` commands 329may be included to update the contents of the branch prior to 330creating the commit. These commands may be supplied in any order. 331However it is recommended that a `filedeleteall` command preceed 332all `filemodify` commands in the same commit, as `filedeleteall` 333wipes the branch clean (see below). 334 335`author` 336^^^^^^^^ 337An `author` command may optionally appear, if the author information 338might differ from the committer information. If `author` is omitted 339then fast-import will automatically use the committer's information for 340the author portion of the commit. See below for a description of 341the fields in `author`, as they are identical to `committer`. 342 343`committer` 344^^^^^^^^^^^ 345The `committer` command indicates who made this commit, and when 346they made it. 347 348Here `<name>` is the person's display name (for example 349``Com M Itter'') and `<email>` is the person's email address 350(``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c) 351and greater-than (\x3e) symbols. These are required to delimit 352the email address from the other fields in the line. Note that 353`<name>` is free-form and may contain any sequence of bytes, except 354`LT` and `LF`. It is typically UTF-8 encoded. 355 356The time of the change is specified by `<when>` using the date format 357that was selected by the \--date-format=<fmt> command line option. 358See ``Date Formats'' above for the set of supported formats, and 359their syntax. 360 361`from` 362^^^^^^ 363The `from` command is used to specify the commit to initialize 364this branch from. This revision will be the first ancestor of the 365new commit. 366 367Omitting the `from` command in the first commit of a new branch 368will cause fast-import to create that commit with no ancestor. This 369tends to be desired only for the initial commit of a project. 370Omitting the `from` command on existing branches is usually desired, 371as the current commit on that branch is automatically assumed to 372be the first ancestor of the new commit. 373 374As `LF` is not valid in a Git refname or SHA-1 expression, no 375quoting or escaping syntax is supported within `<committish>`. 376 377Here `<committish>` is any of the following: 378 379* The name of an existing branch already in fast-import's internal branch 380 table. If fast-import doesn't know the name, its treated as a SHA-1 381 expression. 382 383* A mark reference, `:<idnum>`, where `<idnum>` is the mark number. 384+ 385The reason fast-import uses `:` to denote a mark reference is this character 386is not legal in a Git branch name. The leading `:` makes it easy 387to distingush between the mark 42 (`:42`) and the branch 42 (`42` 388or `refs/heads/42`), or an abbreviated SHA-1 which happened to 389consist only of base-10 digits. 390+ 391Marks must be declared (via `mark`) before they can be used. 392 393* A complete 40 byte or abbreviated commit SHA-1 in hex. 394 395* Any valid Git SHA-1 expression that resolves to a commit. See 396 ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details. 397 398The special case of restarting an incremental import from the 399current branch value should be written as: 400---- 401 from refs/heads/branch^0 402---- 403The `{caret}0` suffix is necessary as fast-import does not permit a branch to 404start from itself, and the branch is created in memory before the 405`from` command is even read from the input. Adding `{caret}0` will force 406fast-import to resolve the commit through Git's revision parsing library, 407rather than its internal branch table, thereby loading in the 408existing value of the branch. 409 410`merge` 411^^^^^^^ 412Includes one additional ancestor commit, and makes the current 413commit a merge commit. An unlimited number of `merge` commands per 414commit are permitted by fast-import, thereby establishing an n-way merge. 415However Git's other tools never create commits with more than 15 416additional ancestors (forming a 16-way merge). For this reason 417it is suggested that frontends do not use more than 15 `merge` 418commands per commit. 419 420Here `<committish>` is any of the commit specification expressions 421also accepted by `from` (see above). 422 423`filemodify` 424^^^^^^^^^^^^ 425Included in a `commit` command to add a new file or change the 426content of an existing file. This command has two different means 427of specifying the content of the file. 428 429External data format:: 430 The data content for the file was already supplied by a prior 431 `blob` command. The frontend just needs to connect it. 432+ 433.... 434 'M' SP <mode> SP <dataref> SP <path> LF 435.... 436+ 437Here `<dataref>` can be either a mark reference (`:<idnum>`) 438set by a prior `blob` command, or a full 40-byte SHA-1 of an 439existing Git blob object. 440 441Inline data format:: 442 The data content for the file has not been supplied yet. 443 The frontend wants to supply it as part of this modify 444 command. 445+ 446.... 447 'M' SP <mode> SP 'inline' SP <path> LF 448 data 449.... 450+ 451See below for a detailed description of the `data` command. 452 453In both formats `<mode>` is the type of file entry, specified 454in octal. Git only supports the following modes: 455 456* `100644` or `644`: A normal (not-executable) file. The majority 457 of files in most projects use this mode. If in doubt, this is 458 what you want. 459* `100755` or `755`: A normal, but executable, file. 460* `120000`: A symlink, the content of the file will be the link target. 461 462In both formats `<path>` is the complete path of the file to be added 463(if not already existing) or modified (if already existing). 464 465A `<path>` string must use UNIX-style directory separators (forward 466slash `/`), may contain any byte other than `LF`, and must not 467start with double quote (`"`). 468 469If an `LF` or double quote must be encoded into `<path>` shell-style 470quoting should be used, e.g. `"path/with\n and \" in it"`. 471 472The value of `<path>` must be in canoncial form. That is it must not: 473 474* contain an empty directory component (e.g. `foo//bar` is invalid), 475* end with a directory separator (e.g. `foo/` is invalid), 476* start with a directory separator (e.g. `/foo` is invalid), 477* contain the special component `.` or `..` (e.g. `foo/./bar` and 478 `foo/../bar` are invalid). 479 480It is recommended that `<path>` always be encoded using UTF-8. 481 482`filedelete` 483^^^^^^^^^^^^ 484Included in a `commit` command to remove a file from the branch. 485If the file removal makes its directory empty, the directory will 486be automatically removed too. This cascades up the tree until the 487first non-empty directory or the root is reached. 488 489.... 490 'D' SP <path> LF 491.... 492 493here `<path>` is the complete path of the file to be removed. 494See `filemodify` above for a detailed description of `<path>`. 495 496`filedeleteall` 497^^^^^^^^^^^^^^^ 498Included in a `commit` command to remove all files (and also all 499directories) from the branch. This command resets the internal 500branch structure to have no files in it, allowing the frontend 501to subsequently add all interesting files from scratch. 502 503.... 504 'deleteall' LF 505.... 506 507This command is extremely useful if the frontend does not know 508(or does not care to know) what files are currently on the branch, 509and therefore cannot generate the proper `filedelete` commands to 510update the content. 511 512Issuing a `filedeleteall` followed by the needed `filemodify` 513commands to set the correct content will produce the same results 514as sending only the needed `filemodify` and `filedelete` commands. 515The `filedeleteall` approach may however require fast-import to use slightly 516more memory per active branch (less than 1 MiB for even most large 517projects); so frontends that can easily obtain only the affected 518paths for a commit are encouraged to do so. 519 520`mark` 521~~~~~~ 522Arranges for fast-import to save a reference to the current object, allowing 523the frontend to recall this object at a future point in time, without 524knowing its SHA-1. Here the current object is the object creation 525command the `mark` command appears within. This can be `commit`, 526`tag`, and `blob`, but `commit` is the most common usage. 527 528.... 529 'mark' SP ':' <idnum> LF 530.... 531 532where `<idnum>` is the number assigned by the frontend to this mark. 533The value of `<idnum>` is expressed as an ASCII decimal integer. 534The value 0 is reserved and cannot be used as 535a mark. Only values greater than or equal to 1 may be used as marks. 536 537New marks are created automatically. Existing marks can be moved 538to another object simply by reusing the same `<idnum>` in another 539`mark` command. 540 541`tag` 542~~~~~ 543Creates an annotated tag referring to a specific commit. To create 544lightweight (non-annotated) tags see the `reset` command below. 545 546.... 547 'tag' SP <name> LF 548 'from' SP <committish> LF 549 'tagger' SP <name> SP LT <email> GT SP <when> LF 550 data 551.... 552 553where `<name>` is the name of the tag to create. 554 555Tag names are automatically prefixed with `refs/tags/` when stored 556in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would 557use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the 558corresponding ref as `refs/tags/RELENG-1_0-FINAL`. 559 560The value of `<name>` must be a valid refname in Git and therefore 561may contain forward slashes. As `LF` is not valid in a Git refname, 562no quoting or escaping syntax is supported here. 563 564The `from` command is the same as in the `commit` command; see 565above for details. 566 567The `tagger` command uses the same format as `committer` within 568`commit`; again see above for details. 569 570The `data` command following `tagger` must supply the annotated tag 571message (see below for `data` command syntax). To import an empty 572tag message use a 0 length data. Tag messages are free-form and are 573not interpreted by Git. Currently they must be encoded in UTF-8, 574as fast-import does not permit other encodings to be specified. 575 576Signing annotated tags during import from within fast-import is not 577supported. Trying to include your own PGP/GPG signature is not 578recommended, as the frontend does not (easily) have access to the 579complete set of bytes which normally goes into such a signature. 580If signing is required, create lightweight tags from within fast-import with 581`reset`, then create the annotated versions of those tags offline 582with the standard gitlink:git-tag[1] process. 583 584`reset` 585~~~~~~~ 586Creates (or recreates) the named branch, optionally starting from 587a specific revision. The reset command allows a frontend to issue 588a new `from` command for an existing branch, or to create a new 589branch from an existing commit without creating a new commit. 590 591.... 592 'reset' SP <ref> LF 593 ('from' SP <committish> LF)? 594 LF 595.... 596 597For a detailed description of `<ref>` and `<committish>` see above 598under `commit` and `from`. 599 600The `reset` command can also be used to create lightweight 601(non-annotated) tags. For example: 602 603==== 604 reset refs/tags/938 605 from :938 606==== 607 608would create the lightweight tag `refs/tags/938` referring to 609whatever commit mark `:938` references. 610 611`blob` 612~~~~~~ 613Requests writing one file revision to the packfile. The revision 614is not connected to any commit; this connection must be formed in 615a subsequent `commit` command by referencing the blob through an 616assigned mark. 617 618.... 619 'blob' LF 620 mark? 621 data 622.... 623 624The mark command is optional here as some frontends have chosen 625to generate the Git SHA-1 for the blob on their own, and feed that 626directly to `commit`. This is typically more work than its worth 627however, as marks are inexpensive to store and easy to use. 628 629`data` 630~~~~~~ 631Supplies raw data (for use as blob/file content, commit messages, or 632annotated tag messages) to fast-import. Data can be supplied using an exact 633byte count or delimited with a terminating line. Real frontends 634intended for production-quality conversions should always use the 635exact byte count format, as it is more robust and performs better. 636The delimited format is intended primarily for testing fast-import. 637 638Exact byte count format:: 639 The frontend must specify the number of bytes of data. 640+ 641.... 642 'data' SP <count> LF 643 <raw> LF 644.... 645+ 646where `<count>` is the exact number of bytes appearing within 647`<raw>`. The value of `<count>` is expressed as an ASCII decimal 648integer. The `LF` on either side of `<raw>` is not 649included in `<count>` and will not be included in the imported data. 650 651Delimited format:: 652 A delimiter string is used to mark the end of the data. 653 fast-import will compute the length by searching for the delimiter. 654 This format is primarly useful for testing and is not 655 recommended for real data. 656+ 657.... 658 'data' SP '<<' <delim> LF 659 <raw> LF 660 <delim> LF 661.... 662+ 663where `<delim>` is the chosen delimiter string. The string `<delim>` 664must not appear on a line by itself within `<raw>`, as otherwise 665fast-import will think the data ends earlier than it really does. The `LF` 666immediately trailing `<raw>` is part of `<raw>`. This is one of 667the limitations of the delimited format, it is impossible to supply 668a data chunk which does not have an LF as its last byte. 669 670`checkpoint` 671~~~~~~~~~~~~ 672Forces fast-import to close the current packfile, start a new one, and to 673save out all current branch refs, tags and marks. 674 675.... 676 'checkpoint' LF 677 LF 678.... 679 680Note that fast-import automatically switches packfiles when the current 681packfile reaches \--max-pack-size, or 4 GiB, whichever limit is 682smaller. During an automatic packfile switch fast-import does not update 683the branch refs, tags or marks. 684 685As a `checkpoint` can require a significant amount of CPU time and 686disk IO (to compute the overall pack SHA-1 checksum, generate the 687corresponding index file, and update the refs) it can easily take 688several minutes for a single `checkpoint` command to complete. 689 690Frontends may choose to issue checkpoints during extremely large 691and long running imports, or when they need to allow another Git 692process access to a branch. However given that a 30 GiB Subversion 693repository can be loaded into Git through fast-import in about 3 hours, 694explicit checkpointing may not be necessary. 695 696 697Tips and Tricks 698--------------- 699The following tips and tricks have been collected from various 700users of fast-import, and are offered here as suggestions. 701 702Use One Mark Per Commit 703~~~~~~~~~~~~~~~~~~~~~~~ 704When doing a repository conversion, use a unique mark per commit 705(`mark :<n>`) and supply the \--export-marks option on the command 706line. fast-import will dump a file which lists every mark and the Git 707object SHA-1 that corresponds to it. If the frontend can tie 708the marks back to the source repository, it is easy to verify the 709accuracy and completeness of the import by comparing each Git 710commit to the corresponding source revision. 711 712Coming from a system such as Perforce or Subversion this should be 713quite simple, as the fast-import mark can also be the Perforce changeset 714number or the Subversion revision number. 715 716Freely Skip Around Branches 717~~~~~~~~~~~~~~~~~~~~~~~~~~~ 718Don't bother trying to optimize the frontend to stick to one branch 719at a time during an import. Although doing so might be slightly 720faster for fast-import, it tends to increase the complexity of the frontend 721code considerably. 722 723The branch LRU builtin to fast-import tends to behave very well, and the 724cost of activating an inactive branch is so low that bouncing around 725between branches has virtually no impact on import performance. 726 727Handling Renames 728~~~~~~~~~~~~~~~~ 729When importing a renamed file or directory, simply delete the old 730name(s) and modify the new name(s) during the corresponding commit. 731Git performs rename detection after-the-fact, rather than explicitly 732during a commit. 733 734Use Tag Fixup Branches 735~~~~~~~~~~~~~~~~~~~~~~ 736Some other SCM systems let the user create a tag from multiple 737files which are not from the same commit/changeset. Or to create 738tags which are a subset of the files available in the repository. 739 740Importing these tags as-is in Git is impossible without making at 741least one commit which ``fixes up'' the files to match the content 742of the tag. Use fast-import's `reset` command to reset a dummy branch 743outside of your normal branch space to the base commit for the tag, 744then commit one or more file fixup commits, and finally tag the 745dummy branch. 746 747For example since all normal branches are stored under `refs/heads/` 748name the tag fixup branch `TAG_FIXUP`. This way it is impossible for 749the fixup branch used by the importer to have namespace conflicts 750with real branches imported from the source (the name `TAG_FIXUP` 751is not `refs/heads/TAG_FIXUP`). 752 753When committing fixups, consider using `merge` to connect the 754commit(s) which are supplying file revisions to the fixup branch. 755Doing so will allow tools such as gitlink:git-blame[1] to track 756through the real commit history and properly annotate the source 757files. 758 759After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP` 760to remove the dummy branch. 761 762Import Now, Repack Later 763~~~~~~~~~~~~~~~~~~~~~~~~ 764As soon as fast-import completes the Git repository is completely valid 765and ready for use. Typicallly this takes only a very short time, 766even for considerably large projects (100,000+ commits). 767 768However repacking the repository is necessary to improve data 769locality and access performance. It can also take hours on extremely 770large projects (especially if -f and a large \--window parameter is 771used). Since repacking is safe to run alongside readers and writers, 772run the repack in the background and let it finish when it finishes. 773There is no reason to wait to explore your new Git project! 774 775If you choose to wait for the repack, don't try to run benchmarks 776or performance tests until repacking is completed. fast-import outputs 777suboptimal packfiles that are simply never seen in real use 778situations. 779 780Repacking Historical Data 781~~~~~~~~~~~~~~~~~~~~~~~~~ 782If you are repacking very old imported data (e.g. older than the 783last year), consider expending some extra CPU time and supplying 784\--window=50 (or higher) when you run gitlink:git-repack[1]. 785This will take longer, but will also produce a smaller packfile. 786You only need to expend the effort once, and everyone using your 787project will benefit from the smaller repository. 788 789 790Packfile Optimization 791--------------------- 792When packing a blob fast-import always attempts to deltify against the last 793blob written. Unless specifically arranged for by the frontend, 794this will probably not be a prior version of the same file, so the 795generated delta will not be the smallest possible. The resulting 796packfile will be compressed, but will not be optimal. 797 798Frontends which have efficient access to all revisions of a 799single file (for example reading an RCS/CVS ,v file) can choose 800to supply all revisions of that file as a sequence of consecutive 801`blob` commands. This allows fast-import to deltify the different file 802revisions against each other, saving space in the final packfile. 803Marks can be used to later identify individual file revisions during 804a sequence of `commit` commands. 805 806The packfile(s) created by fast-import do not encourage good disk access 807patterns. This is caused by fast-import writing the data in the order 808it is received on standard input, while Git typically organizes 809data within packfiles to make the most recent (current tip) data 810appear before historical data. Git also clusters commits together, 811speeding up revision traversal through better cache locality. 812 813For this reason it is strongly recommended that users repack the 814repository with `git repack -a -d` after fast-import completes, allowing 815Git to reorganize the packfiles for faster data access. If blob 816deltas are suboptimal (see above) then also adding the `-f` option 817to force recomputation of all deltas can significantly reduce the 818final packfile size (30-50% smaller can be quite typical). 819 820 821Memory Utilization 822------------------ 823There are a number of factors which affect how much memory fast-import 824requires to perform an import. Like critical sections of core 825Git, fast-import uses its own memory allocators to ammortize any overheads 826associated with malloc. In practice fast-import tends to ammoritize any 827malloc overheads to 0, due to its use of large block allocations. 828 829per object 830~~~~~~~~~~ 831fast-import maintains an in-memory structure for every object written in 832this execution. On a 32 bit system the structure is 32 bytes, 833on a 64 bit system the structure is 40 bytes (due to the larger 834pointer sizes). Objects in the table are not deallocated until 835fast-import terminates. Importing 2 million objects on a 32 bit system 836will require approximately 64 MiB of memory. 837 838The object table is actually a hashtable keyed on the object name 839(the unique SHA-1). This storage configuration allows fast-import to reuse 840an existing or already written object and avoid writing duplicates 841to the output packfile. Duplicate blobs are surprisingly common 842in an import, typically due to branch merges in the source. 843 844per mark 845~~~~~~~~ 846Marks are stored in a sparse array, using 1 pointer (4 bytes or 8 847bytes, depending on pointer size) per mark. Although the array 848is sparse, frontends are still strongly encouraged to use marks 849between 1 and n, where n is the total number of marks required for 850this import. 851 852per branch 853~~~~~~~~~~ 854Branches are classified as active and inactive. The memory usage 855of the two classes is significantly different. 856 857Inactive branches are stored in a structure which uses 96 or 120 858bytes (32 bit or 64 bit systems, respectively), plus the length of 859the branch name (typically under 200 bytes), per branch. fast-import will 860easily handle as many as 10,000 inactive branches in under 2 MiB 861of memory. 862 863Active branches have the same overhead as inactive branches, but 864also contain copies of every tree that has been recently modified on 865that branch. If subtree `include` has not been modified since the 866branch became active, its contents will not be loaded into memory, 867but if subtree `src` has been modified by a commit since the branch 868became active, then its contents will be loaded in memory. 869 870As active branches store metadata about the files contained on that 871branch, their in-memory storage size can grow to a considerable size 872(see below). 873 874fast-import automatically moves active branches to inactive status based on 875a simple least-recently-used algorithm. The LRU chain is updated on 876each `commit` command. The maximum number of active branches can be 877increased or decreased on the command line with \--active-branches=. 878 879per active tree 880~~~~~~~~~~~~~~~ 881Trees (aka directories) use just 12 bytes of memory on top of the 882memory required for their entries (see ``per active file'' below). 883The cost of a tree is virtually 0, as its overhead ammortizes out 884over the individual file entries. 885 886per active file entry 887~~~~~~~~~~~~~~~~~~~~~ 888Files (and pointers to subtrees) within active trees require 52 or 64 889bytes (32/64 bit platforms) per entry. To conserve space, file and 890tree names are pooled in a common string table, allowing the filename 891``Makefile'' to use just 16 bytes (after including the string header 892overhead) no matter how many times it occurs within the project. 893 894The active branch LRU, when coupled with the filename string pool 895and lazy loading of subtrees, allows fast-import to efficiently import 896projects with 2,000+ branches and 45,114+ files in a very limited 897memory footprint (less than 2.7 MiB per active branch). 898 899 900Author 901------ 902Written by Shawn O. Pearce <spearce@spearce.org>. 903 904Documentation 905-------------- 906Documentation by Shawn O. Pearce <spearce@spearce.org>. 907 908GIT 909--- 910Part of the gitlink:git[7] suite 911