1git-fast-import(1) 2================== 3 4NAME 5---- 6git-fast-import - Backend for fast Git data importers 7 8 9SYNOPSIS 10-------- 11[verse] 12frontend | 'git fast-import' [<options>] 13 14DESCRIPTION 15----------- 16This program is usually not what the end user wants to run directly. 17Most end users want to use one of the existing frontend programs, 18which parses a specific type of foreign source and feeds the contents 19stored there to 'git fast-import'. 20 21fast-import reads a mixed command/data stream from standard input and 22writes one or more packfiles directly into the current repository. 23When EOF is received on standard input, fast import writes out 24updated branch and tag refs, fully updating the current repository 25with the newly imported data. 26 27The fast-import backend itself can import into an empty repository (one that 28has already been initialized by 'git init') or incrementally 29update an existing populated repository. Whether or not incremental 30imports are supported from a particular foreign source depends on 31the frontend program in use. 32 33 34OPTIONS 35------- 36 37--force:: 38 Force updating modified existing branches, even if doing 39 so would cause commits to be lost (as the new commit does 40 not contain the old commit). 41 42--quiet:: 43 Disable the output shown by --stats, making fast-import usually 44 be silent when it is successful. However, if the import stream 45 has directives intended to show user output (e.g. `progress` 46 directives), the corresponding messages will still be shown. 47 48--stats:: 49 Display some basic statistics about the objects fast-import has 50 created, the packfiles they were stored into, and the 51 memory used by fast-import during this run. Showing this output 52 is currently the default, but can be disabled with --quiet. 53 54Options for Frontends 55~~~~~~~~~~~~~~~~~~~~~ 56 57--cat-blob-fd=<fd>:: 58 Write responses to `get-mark`, `cat-blob`, and `ls` queries to the 59 file descriptor <fd> instead of `stdout`. Allows `progress` 60 output intended for the end-user to be separated from other 61 output. 62 63--date-format=<fmt>:: 64 Specify the type of dates the frontend will supply to 65 fast-import within `author`, `committer` and `tagger` commands. 66 See ``Date Formats'' below for details about which formats 67 are supported, and their syntax. 68 69--done:: 70 Terminate with error if there is no `done` command at the end of 71 the stream. This option might be useful for detecting errors 72 that cause the frontend to terminate before it has started to 73 write a stream. 74 75Locations of Marks Files 76~~~~~~~~~~~~~~~~~~~~~~~~ 77 78--export-marks=<file>:: 79 Dumps the internal marks table to <file> when complete. 80 Marks are written one per line as `:markid SHA-1`. 81 Frontends can use this file to validate imports after they 82 have been completed, or to save the marks table across 83 incremental runs. As <file> is only opened and truncated 84 at checkpoint (or completion) the same path can also be 85 safely given to --import-marks. 86 87--import-marks=<file>:: 88 Before processing any input, load the marks specified in 89 <file>. The input file must exist, must be readable, and 90 must use the same format as produced by --export-marks. 91 Multiple options may be supplied to import more than one 92 set of marks. If a mark is defined to different values, 93 the last file wins. 94 95--import-marks-if-exists=<file>:: 96 Like --import-marks but instead of erroring out, silently 97 skips the file if it does not exist. 98 99--[no-]relative-marks:: 100 After specifying --relative-marks the paths specified 101 with --import-marks= and --export-marks= are relative 102 to an internal directory in the current repository. 103 In git-fast-import this means that the paths are relative 104 to the .git/info/fast-import directory. However, other 105 importers may use a different location. 106+ 107Relative and non-relative marks may be combined by interweaving 108--(no-)-relative-marks with the --(import|export)-marks= options. 109 110Performance and Compression Tuning 111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 112 113--active-branches=<n>:: 114 Maximum number of branches to maintain active at once. 115 See ``Memory Utilization'' below for details. Default is 5. 116 117--big-file-threshold=<n>:: 118 Maximum size of a blob that fast-import will attempt to 119 create a delta for, expressed in bytes. The default is 512m 120 (512 MiB). Some importers may wish to lower this on systems 121 with constrained memory. 122 123--depth=<n>:: 124 Maximum delta depth, for blob and tree deltification. 125 Default is 50. 126 127--export-pack-edges=<file>:: 128 After creating a packfile, print a line of data to 129 <file> listing the filename of the packfile and the last 130 commit on each branch that was written to that packfile. 131 This information may be useful after importing projects 132 whose total object set exceeds the 4 GiB packfile limit, 133 as these commits can be used as edge points during calls 134 to 'git pack-objects'. 135 136--max-pack-size=<n>:: 137 Maximum size of each output packfile. 138 The default is unlimited. 139 140fastimport.unpackLimit:: 141 See linkgit:git-config[1] 142 143PERFORMANCE 144----------- 145The design of fast-import allows it to import large projects in a minimum 146amount of memory usage and processing time. Assuming the frontend 147is able to keep up with fast-import and feed it a constant stream of data, 148import times for projects holding 10+ years of history and containing 149100,000+ individual commits are generally completed in just 1-2 150hours on quite modest (~$2,000 USD) hardware. 151 152Most bottlenecks appear to be in foreign source data access (the 153source just cannot extract revisions fast enough) or disk IO (fast-import 154writes as fast as the disk will take the data). Imports will run 155faster if the source data is stored on a different drive than the 156destination Git repository (due to less IO contention). 157 158 159DEVELOPMENT COST 160---------------- 161A typical frontend for fast-import tends to weigh in at approximately 200 162lines of Perl/Python/Ruby code. Most developers have been able to 163create working importers in just a couple of hours, even though it 164is their first exposure to fast-import, and sometimes even to Git. This is 165an ideal situation, given that most conversion tools are throw-away 166(use once, and never look back). 167 168 169PARALLEL OPERATION 170------------------ 171Like 'git push' or 'git fetch', imports handled by fast-import are safe to 172run alongside parallel `git repack -a -d` or `git gc` invocations, 173or any other Git operation (including 'git prune', as loose objects 174are never used by fast-import). 175 176fast-import does not lock the branch or tag refs it is actively importing. 177After the import, during its ref update phase, fast-import tests each 178existing branch ref to verify the update will be a fast-forward 179update (the commit stored in the ref is contained in the new 180history of the commit to be written). If the update is not a 181fast-forward update, fast-import will skip updating that ref and instead 182prints a warning message. fast-import will always attempt to update all 183branch refs, and does not stop on the first failure. 184 185Branch updates can be forced with --force, but it's recommended that 186this only be used on an otherwise quiet repository. Using --force 187is not necessary for an initial import into an empty repository. 188 189 190TECHNICAL DISCUSSION 191-------------------- 192fast-import tracks a set of branches in memory. Any branch can be created 193or modified at any point during the import process by sending a 194`commit` command on the input stream. This design allows a frontend 195program to process an unlimited number of branches simultaneously, 196generating commits in the order they are available from the source 197data. It also simplifies the frontend programs considerably. 198 199fast-import does not use or alter the current working directory, or any 200file within it. (It does however update the current Git repository, 201as referenced by `GIT_DIR`.) Therefore an import frontend may use 202the working directory for its own purposes, such as extracting file 203revisions from the foreign source. This ignorance of the working 204directory also allows fast-import to run very quickly, as it does not 205need to perform any costly file update operations when switching 206between branches. 207 208INPUT FORMAT 209------------ 210With the exception of raw file data (which Git does not interpret) 211the fast-import input format is text (ASCII) based. This text based 212format simplifies development and debugging of frontend programs, 213especially when a higher level language such as Perl, Python or 214Ruby is being used. 215 216fast-import is very strict about its input. Where we say SP below we mean 217*exactly* one space. Likewise LF means one (and only one) linefeed 218and HT one (and only one) horizontal tab. 219Supplying additional whitespace characters will cause unexpected 220results, such as branch names or file names with leading or trailing 221spaces in their name, or early termination of fast-import when it encounters 222unexpected input. 223 224Stream Comments 225~~~~~~~~~~~~~~~ 226To aid in debugging frontends fast-import ignores any line that 227begins with `#` (ASCII pound/hash) up to and including the line 228ending `LF`. A comment line may contain any sequence of bytes 229that does not contain an LF and therefore may be used to include 230any detailed debugging information that might be specific to the 231frontend and useful when inspecting a fast-import data stream. 232 233Date Formats 234~~~~~~~~~~~~ 235The following date formats are supported. A frontend should select 236the format it will use for this import by passing the format name 237in the --date-format=<fmt> command-line option. 238 239`raw`:: 240 This is the Git native format and is `<time> SP <offutc>`. 241 It is also fast-import's default format, if --date-format was 242 not specified. 243+ 244The time of the event is specified by `<time>` as the number of 245seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is 246written as an ASCII decimal integer. 247+ 248The local offset is specified by `<offutc>` as a positive or negative 249offset from UTC. For example EST (which is 5 hours behind UTC) 250would be expressed in `<tz>` by ``-0500'' while UTC is ``+0000''. 251The local offset does not affect `<time>`; it is used only as an 252advisement to help formatting routines display the timestamp. 253+ 254If the local offset is not available in the source material, use 255``+0000'', or the most common local offset. For example many 256organizations have a CVS repository which has only ever been accessed 257by users who are located in the same location and time zone. In this 258case a reasonable offset from UTC could be assumed. 259+ 260Unlike the `rfc2822` format, this format is very strict. Any 261variation in formatting will cause fast-import to reject the value. 262 263`rfc2822`:: 264 This is the standard email format as described by RFC 2822. 265+ 266An example value is ``Tue Feb 6 11:22:18 2007 -0500''. The Git 267parser is accurate, but a little on the lenient side. It is the 268same parser used by 'git am' when applying patches 269received from email. 270+ 271Some malformed strings may be accepted as valid dates. In some of 272these cases Git will still be able to obtain the correct date from 273the malformed string. There are also some types of malformed 274strings which Git will parse wrong, and yet consider valid. 275Seriously malformed strings will be rejected. 276+ 277Unlike the `raw` format above, the time zone/UTC offset information 278contained in an RFC 2822 date string is used to adjust the date 279value to UTC prior to storage. Therefore it is important that 280this information be as accurate as possible. 281+ 282If the source material uses RFC 2822 style dates, 283the frontend should let fast-import handle the parsing and conversion 284(rather than attempting to do it itself) as the Git parser has 285been well tested in the wild. 286+ 287Frontends should prefer the `raw` format if the source material 288already uses UNIX-epoch format, can be coaxed to give dates in that 289format, or its format is easily convertible to it, as there is no 290ambiguity in parsing. 291 292`now`:: 293 Always use the current time and time zone. The literal 294 `now` must always be supplied for `<when>`. 295+ 296This is a toy format. The current time and time zone of this system 297is always copied into the identity string at the time it is being 298created by fast-import. There is no way to specify a different time or 299time zone. 300+ 301This particular format is supplied as it's short to implement and 302may be useful to a process that wants to create a new commit 303right now, without needing to use a working directory or 304'git update-index'. 305+ 306If separate `author` and `committer` commands are used in a `commit` 307the timestamps may not match, as the system clock will be polled 308twice (once for each command). The only way to ensure that both 309author and committer identity information has the same timestamp 310is to omit `author` (thus copying from `committer`) or to use a 311date format other than `now`. 312 313Commands 314~~~~~~~~ 315fast-import accepts several commands to update the current repository 316and control the current import process. More detailed discussion 317(with examples) of each command follows later. 318 319`commit`:: 320 Creates a new branch or updates an existing branch by 321 creating a new commit and updating the branch to point at 322 the newly created commit. 323 324`tag`:: 325 Creates an annotated tag object from an existing commit or 326 branch. Lightweight tags are not supported by this command, 327 as they are not recommended for recording meaningful points 328 in time. 329 330`reset`:: 331 Reset an existing branch (or a new branch) to a specific 332 revision. This command must be used to change a branch to 333 a specific revision without making a commit on it. 334 335`blob`:: 336 Convert raw file data into a blob, for future use in a 337 `commit` command. This command is optional and is not 338 needed to perform an import. 339 340`checkpoint`:: 341 Forces fast-import to close the current packfile, generate its 342 unique SHA-1 checksum and index, and start a new packfile. 343 This command is optional and is not needed to perform 344 an import. 345 346`progress`:: 347 Causes fast-import to echo the entire line to its own 348 standard output. This command is optional and is not needed 349 to perform an import. 350 351`done`:: 352 Marks the end of the stream. This command is optional 353 unless the `done` feature was requested using the 354 `--done` command-line option or `feature done` command. 355 356`get-mark`:: 357 Causes fast-import to print the SHA-1 corresponding to a mark 358 to the file descriptor set with `--cat-blob-fd`, or `stdout` if 359 unspecified. 360 361`cat-blob`:: 362 Causes fast-import to print a blob in 'cat-file --batch' 363 format to the file descriptor set with `--cat-blob-fd` or 364 `stdout` if unspecified. 365 366`ls`:: 367 Causes fast-import to print a line describing a directory 368 entry in 'ls-tree' format to the file descriptor set with 369 `--cat-blob-fd` or `stdout` if unspecified. 370 371`feature`:: 372 Enable the specified feature. This requires that fast-import 373 supports the specified feature, and aborts if it does not. 374 375`option`:: 376 Specify any of the options listed under OPTIONS that do not 377 change stream semantic to suit the frontend's needs. This 378 command is optional and is not needed to perform an import. 379 380`commit` 381~~~~~~~~ 382Create or update a branch with a new commit, recording one logical 383change to the project. 384 385.... 386 'commit' SP <ref> LF 387 mark? 388 original-oid? 389 ('author' (SP <name>)? SP LT <email> GT SP <when> LF)? 390 'committer' (SP <name>)? SP LT <email> GT SP <when> LF 391 ('encoding' SP <encoding>)? 392 data 393 ('from' SP <commit-ish> LF)? 394 ('merge' SP <commit-ish> LF)? 395 (filemodify | filedelete | filecopy | filerename | filedeleteall | notemodify)* 396 LF? 397.... 398 399where `<ref>` is the name of the branch to make the commit on. 400Typically branch names are prefixed with `refs/heads/` in 401Git, so importing the CVS branch symbol `RELENG-1_0` would use 402`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of 403`<ref>` must be a valid refname in Git. As `LF` is not valid in 404a Git refname, no quoting or escaping syntax is supported here. 405 406A `mark` command may optionally appear, requesting fast-import to save a 407reference to the newly created commit for future use by the frontend 408(see below for format). It is very common for frontends to mark 409every commit they create, thereby allowing future branch creation 410from any imported commit. 411 412The `data` command following `committer` must supply the commit 413message (see below for `data` command syntax). To import an empty 414commit message use a 0 length data. Commit messages are free-form 415and are not interpreted by Git. Currently they must be encoded in 416UTF-8, as fast-import does not permit other encodings to be specified. 417 418Zero or more `filemodify`, `filedelete`, `filecopy`, `filerename`, 419`filedeleteall` and `notemodify` commands 420may be included to update the contents of the branch prior to 421creating the commit. These commands may be supplied in any order. 422However it is recommended that a `filedeleteall` command precede 423all `filemodify`, `filecopy`, `filerename` and `notemodify` commands in 424the same commit, as `filedeleteall` wipes the branch clean (see below). 425 426The `LF` after the command is optional (it used to be required). Note 427that for reasons of backward compatibility, if the commit ends with a 428`data` command (i.e. it has no `from`, `merge`, `filemodify`, 429`filedelete`, `filecopy`, `filerename`, `filedeleteall` or 430`notemodify` commands) then two `LF` commands may appear at the end of 431the command instead of just one. 432 433`author` 434^^^^^^^^ 435An `author` command may optionally appear, if the author information 436might differ from the committer information. If `author` is omitted 437then fast-import will automatically use the committer's information for 438the author portion of the commit. See below for a description of 439the fields in `author`, as they are identical to `committer`. 440 441`committer` 442^^^^^^^^^^^ 443The `committer` command indicates who made this commit, and when 444they made it. 445 446Here `<name>` is the person's display name (for example 447``Com M Itter'') and `<email>` is the person's email address 448(``\cm@example.com''). `LT` and `GT` are the literal less-than (\x3c) 449and greater-than (\x3e) symbols. These are required to delimit 450the email address from the other fields in the line. Note that 451`<name>` and `<email>` are free-form and may contain any sequence 452of bytes, except `LT`, `GT` and `LF`. `<name>` is typically UTF-8 encoded. 453 454The time of the change is specified by `<when>` using the date format 455that was selected by the --date-format=<fmt> command-line option. 456See ``Date Formats'' above for the set of supported formats, and 457their syntax. 458 459`encoding` 460^^^^^^^^^^ 461The optional `encoding` command indicates the encoding of the commit 462message. Most commits are UTF-8 and the encoding is omitted, but this 463allows importing commit messages into git without first reencoding them. 464 465`from` 466^^^^^^ 467The `from` command is used to specify the commit to initialize 468this branch from. This revision will be the first ancestor of the 469new commit. The state of the tree built at this commit will begin 470with the state at the `from` commit, and be altered by the content 471modifications in this commit. 472 473Omitting the `from` command in the first commit of a new branch 474will cause fast-import to create that commit with no ancestor. This 475tends to be desired only for the initial commit of a project. 476If the frontend creates all files from scratch when making a new 477branch, a `merge` command may be used instead of `from` to start 478the commit with an empty tree. 479Omitting the `from` command on existing branches is usually desired, 480as the current commit on that branch is automatically assumed to 481be the first ancestor of the new commit. 482 483As `LF` is not valid in a Git refname or SHA-1 expression, no 484quoting or escaping syntax is supported within `<commit-ish>`. 485 486Here `<commit-ish>` is any of the following: 487 488* The name of an existing branch already in fast-import's internal branch 489 table. If fast-import doesn't know the name, it's treated as a SHA-1 490 expression. 491 492* A mark reference, `:<idnum>`, where `<idnum>` is the mark number. 493+ 494The reason fast-import uses `:` to denote a mark reference is this character 495is not legal in a Git branch name. The leading `:` makes it easy 496to distinguish between the mark 42 (`:42`) and the branch 42 (`42` 497or `refs/heads/42`), or an abbreviated SHA-1 which happened to 498consist only of base-10 digits. 499+ 500Marks must be declared (via `mark`) before they can be used. 501 502* A complete 40 byte or abbreviated commit SHA-1 in hex. 503 504* Any valid Git SHA-1 expression that resolves to a commit. See 505 ``SPECIFYING REVISIONS'' in linkgit:gitrevisions[7] for details. 506 507* The special null SHA-1 (40 zeros) specifies that the branch is to be 508 removed. 509 510The special case of restarting an incremental import from the 511current branch value should be written as: 512---- 513 from refs/heads/branch^0 514---- 515The `^0` suffix is necessary as fast-import does not permit a branch to 516start from itself, and the branch is created in memory before the 517`from` command is even read from the input. Adding `^0` will force 518fast-import to resolve the commit through Git's revision parsing library, 519rather than its internal branch table, thereby loading in the 520existing value of the branch. 521 522`merge` 523^^^^^^^ 524Includes one additional ancestor commit. The additional ancestry 525link does not change the way the tree state is built at this commit. 526If the `from` command is 527omitted when creating a new branch, the first `merge` commit will be 528the first ancestor of the current commit, and the branch will start 529out with no files. An unlimited number of `merge` commands per 530commit are permitted by fast-import, thereby establishing an n-way merge. 531 532Here `<commit-ish>` is any of the commit specification expressions 533also accepted by `from` (see above). 534 535`filemodify` 536^^^^^^^^^^^^ 537Included in a `commit` command to add a new file or change the 538content of an existing file. This command has two different means 539of specifying the content of the file. 540 541External data format:: 542 The data content for the file was already supplied by a prior 543 `blob` command. The frontend just needs to connect it. 544+ 545.... 546 'M' SP <mode> SP <dataref> SP <path> LF 547.... 548+ 549Here usually `<dataref>` must be either a mark reference (`:<idnum>`) 550set by a prior `blob` command, or a full 40-byte SHA-1 of an 551existing Git blob object. If `<mode>` is `040000`` then 552`<dataref>` must be the full 40-byte SHA-1 of an existing 553Git tree object or a mark reference set with `--import-marks`. 554 555Inline data format:: 556 The data content for the file has not been supplied yet. 557 The frontend wants to supply it as part of this modify 558 command. 559+ 560.... 561 'M' SP <mode> SP 'inline' SP <path> LF 562 data 563.... 564+ 565See below for a detailed description of the `data` command. 566 567In both formats `<mode>` is the type of file entry, specified 568in octal. Git only supports the following modes: 569 570* `100644` or `644`: A normal (not-executable) file. The majority 571 of files in most projects use this mode. If in doubt, this is 572 what you want. 573* `100755` or `755`: A normal, but executable, file. 574* `120000`: A symlink, the content of the file will be the link target. 575* `160000`: A gitlink, SHA-1 of the object refers to a commit in 576 another repository. Git links can only be specified by SHA or through 577 a commit mark. They are used to implement submodules. 578* `040000`: A subdirectory. Subdirectories can only be specified by 579 SHA or through a tree mark set with `--import-marks`. 580 581In both formats `<path>` is the complete path of the file to be added 582(if not already existing) or modified (if already existing). 583 584A `<path>` string must use UNIX-style directory separators (forward 585slash `/`), may contain any byte other than `LF`, and must not 586start with double quote (`"`). 587 588A path can use C-style string quoting; this is accepted in all cases 589and mandatory if the filename starts with double quote or contains 590`LF`. In C-style quoting, the complete name should be surrounded with 591double quotes, and any `LF`, backslash, or double quote characters 592must be escaped by preceding them with a backslash (e.g., 593`"path/with\n, \\ and \" in it"`). 594 595The value of `<path>` must be in canonical form. That is it must not: 596 597* contain an empty directory component (e.g. `foo//bar` is invalid), 598* end with a directory separator (e.g. `foo/` is invalid), 599* start with a directory separator (e.g. `/foo` is invalid), 600* contain the special component `.` or `..` (e.g. `foo/./bar` and 601 `foo/../bar` are invalid). 602 603The root of the tree can be represented by an empty string as `<path>`. 604 605It is recommended that `<path>` always be encoded using UTF-8. 606 607`filedelete` 608^^^^^^^^^^^^ 609Included in a `commit` command to remove a file or recursively 610delete an entire directory from the branch. If the file or directory 611removal makes its parent directory empty, the parent directory will 612be automatically removed too. This cascades up the tree until the 613first non-empty directory or the root is reached. 614 615.... 616 'D' SP <path> LF 617.... 618 619here `<path>` is the complete path of the file or subdirectory to 620be removed from the branch. 621See `filemodify` above for a detailed description of `<path>`. 622 623`filecopy` 624^^^^^^^^^^ 625Recursively copies an existing file or subdirectory to a different 626location within the branch. The existing file or directory must 627exist. If the destination exists it will be completely replaced 628by the content copied from the source. 629 630.... 631 'C' SP <path> SP <path> LF 632.... 633 634here the first `<path>` is the source location and the second 635`<path>` is the destination. See `filemodify` above for a detailed 636description of what `<path>` may look like. To use a source path 637that contains SP the path must be quoted. 638 639A `filecopy` command takes effect immediately. Once the source 640location has been copied to the destination any future commands 641applied to the source location will not impact the destination of 642the copy. 643 644`filerename` 645^^^^^^^^^^^^ 646Renames an existing file or subdirectory to a different location 647within the branch. The existing file or directory must exist. If 648the destination exists it will be replaced by the source directory. 649 650.... 651 'R' SP <path> SP <path> LF 652.... 653 654here the first `<path>` is the source location and the second 655`<path>` is the destination. See `filemodify` above for a detailed 656description of what `<path>` may look like. To use a source path 657that contains SP the path must be quoted. 658 659A `filerename` command takes effect immediately. Once the source 660location has been renamed to the destination any future commands 661applied to the source location will create new files there and not 662impact the destination of the rename. 663 664Note that a `filerename` is the same as a `filecopy` followed by a 665`filedelete` of the source location. There is a slight performance 666advantage to using `filerename`, but the advantage is so small 667that it is never worth trying to convert a delete/add pair in 668source material into a rename for fast-import. This `filerename` 669command is provided just to simplify frontends that already have 670rename information and don't want bother with decomposing it into a 671`filecopy` followed by a `filedelete`. 672 673`filedeleteall` 674^^^^^^^^^^^^^^^ 675Included in a `commit` command to remove all files (and also all 676directories) from the branch. This command resets the internal 677branch structure to have no files in it, allowing the frontend 678to subsequently add all interesting files from scratch. 679 680.... 681 'deleteall' LF 682.... 683 684This command is extremely useful if the frontend does not know 685(or does not care to know) what files are currently on the branch, 686and therefore cannot generate the proper `filedelete` commands to 687update the content. 688 689Issuing a `filedeleteall` followed by the needed `filemodify` 690commands to set the correct content will produce the same results 691as sending only the needed `filemodify` and `filedelete` commands. 692The `filedeleteall` approach may however require fast-import to use slightly 693more memory per active branch (less than 1 MiB for even most large 694projects); so frontends that can easily obtain only the affected 695paths for a commit are encouraged to do so. 696 697`notemodify` 698^^^^^^^^^^^^ 699Included in a `commit` `<notes_ref>` command to add a new note 700annotating a `<commit-ish>` or change this annotation contents. 701Internally it is similar to filemodify 100644 on `<commit-ish>` 702path (maybe split into subdirectories). It's not advised to 703use any other commands to write to the `<notes_ref>` tree except 704`filedeleteall` to delete all existing notes in this tree. 705This command has two different means of specifying the content 706of the note. 707 708External data format:: 709 The data content for the note was already supplied by a prior 710 `blob` command. The frontend just needs to connect it to the 711 commit that is to be annotated. 712+ 713.... 714 'N' SP <dataref> SP <commit-ish> LF 715.... 716+ 717Here `<dataref>` can be either a mark reference (`:<idnum>`) 718set by a prior `blob` command, or a full 40-byte SHA-1 of an 719existing Git blob object. 720 721Inline data format:: 722 The data content for the note has not been supplied yet. 723 The frontend wants to supply it as part of this modify 724 command. 725+ 726.... 727 'N' SP 'inline' SP <commit-ish> LF 728 data 729.... 730+ 731See below for a detailed description of the `data` command. 732 733In both formats `<commit-ish>` is any of the commit specification 734expressions also accepted by `from` (see above). 735 736`mark` 737~~~~~~ 738Arranges for fast-import to save a reference to the current object, allowing 739the frontend to recall this object at a future point in time, without 740knowing its SHA-1. Here the current object is the object creation 741command the `mark` command appears within. This can be `commit`, 742`tag`, and `blob`, but `commit` is the most common usage. 743 744.... 745 'mark' SP ':' <idnum> LF 746.... 747 748where `<idnum>` is the number assigned by the frontend to this mark. 749The value of `<idnum>` is expressed as an ASCII decimal integer. 750The value 0 is reserved and cannot be used as 751a mark. Only values greater than or equal to 1 may be used as marks. 752 753New marks are created automatically. Existing marks can be moved 754to another object simply by reusing the same `<idnum>` in another 755`mark` command. 756 757`original-oid` 758~~~~~~~~~~~~~~ 759Provides the name of the object in the original source control system. 760fast-import will simply ignore this directive, but filter processes 761which operate on and modify the stream before feeding to fast-import 762may have uses for this information 763 764.... 765 'original-oid' SP <object-identifier> LF 766.... 767 768where `<object-identifer>` is any string not containing LF. 769 770`tag` 771~~~~~ 772Creates an annotated tag referring to a specific commit. To create 773lightweight (non-annotated) tags see the `reset` command below. 774 775.... 776 'tag' SP <name> LF 777 'from' SP <commit-ish> LF 778 original-oid? 779 'tagger' (SP <name>)? SP LT <email> GT SP <when> LF 780 data 781.... 782 783where `<name>` is the name of the tag to create. 784 785Tag names are automatically prefixed with `refs/tags/` when stored 786in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would 787use just `RELENG-1_0-FINAL` for `<name>`, and fast-import will write the 788corresponding ref as `refs/tags/RELENG-1_0-FINAL`. 789 790The value of `<name>` must be a valid refname in Git and therefore 791may contain forward slashes. As `LF` is not valid in a Git refname, 792no quoting or escaping syntax is supported here. 793 794The `from` command is the same as in the `commit` command; see 795above for details. 796 797The `tagger` command uses the same format as `committer` within 798`commit`; again see above for details. 799 800The `data` command following `tagger` must supply the annotated tag 801message (see below for `data` command syntax). To import an empty 802tag message use a 0 length data. Tag messages are free-form and are 803not interpreted by Git. Currently they must be encoded in UTF-8, 804as fast-import does not permit other encodings to be specified. 805 806Signing annotated tags during import from within fast-import is not 807supported. Trying to include your own PGP/GPG signature is not 808recommended, as the frontend does not (easily) have access to the 809complete set of bytes which normally goes into such a signature. 810If signing is required, create lightweight tags from within fast-import with 811`reset`, then create the annotated versions of those tags offline 812with the standard 'git tag' process. 813 814`reset` 815~~~~~~~ 816Creates (or recreates) the named branch, optionally starting from 817a specific revision. The reset command allows a frontend to issue 818a new `from` command for an existing branch, or to create a new 819branch from an existing commit without creating a new commit. 820 821.... 822 'reset' SP <ref> LF 823 ('from' SP <commit-ish> LF)? 824 LF? 825.... 826 827For a detailed description of `<ref>` and `<commit-ish>` see above 828under `commit` and `from`. 829 830The `LF` after the command is optional (it used to be required). 831 832The `reset` command can also be used to create lightweight 833(non-annotated) tags. For example: 834 835==== 836 reset refs/tags/938 837 from :938 838==== 839 840would create the lightweight tag `refs/tags/938` referring to 841whatever commit mark `:938` references. 842 843`blob` 844~~~~~~ 845Requests writing one file revision to the packfile. The revision 846is not connected to any commit; this connection must be formed in 847a subsequent `commit` command by referencing the blob through an 848assigned mark. 849 850.... 851 'blob' LF 852 mark? 853 original-oid? 854 data 855.... 856 857The mark command is optional here as some frontends have chosen 858to generate the Git SHA-1 for the blob on their own, and feed that 859directly to `commit`. This is typically more work than it's worth 860however, as marks are inexpensive to store and easy to use. 861 862`data` 863~~~~~~ 864Supplies raw data (for use as blob/file content, commit messages, or 865annotated tag messages) to fast-import. Data can be supplied using an exact 866byte count or delimited with a terminating line. Real frontends 867intended for production-quality conversions should always use the 868exact byte count format, as it is more robust and performs better. 869The delimited format is intended primarily for testing fast-import. 870 871Comment lines appearing within the `<raw>` part of `data` commands 872are always taken to be part of the body of the data and are therefore 873never ignored by fast-import. This makes it safe to import any 874file/message content whose lines might start with `#`. 875 876Exact byte count format:: 877 The frontend must specify the number of bytes of data. 878+ 879.... 880 'data' SP <count> LF 881 <raw> LF? 882.... 883+ 884where `<count>` is the exact number of bytes appearing within 885`<raw>`. The value of `<count>` is expressed as an ASCII decimal 886integer. The `LF` on either side of `<raw>` is not 887included in `<count>` and will not be included in the imported data. 888+ 889The `LF` after `<raw>` is optional (it used to be required) but 890recommended. Always including it makes debugging a fast-import 891stream easier as the next command always starts in column 0 892of the next line, even if `<raw>` did not end with an `LF`. 893 894Delimited format:: 895 A delimiter string is used to mark the end of the data. 896 fast-import will compute the length by searching for the delimiter. 897 This format is primarily useful for testing and is not 898 recommended for real data. 899+ 900.... 901 'data' SP '<<' <delim> LF 902 <raw> LF 903 <delim> LF 904 LF? 905.... 906+ 907where `<delim>` is the chosen delimiter string. The string `<delim>` 908must not appear on a line by itself within `<raw>`, as otherwise 909fast-import will think the data ends earlier than it really does. The `LF` 910immediately trailing `<raw>` is part of `<raw>`. This is one of 911the limitations of the delimited format, it is impossible to supply 912a data chunk which does not have an LF as its last byte. 913+ 914The `LF` after `<delim> LF` is optional (it used to be required). 915 916`checkpoint` 917~~~~~~~~~~~~ 918Forces fast-import to close the current packfile, start a new one, and to 919save out all current branch refs, tags and marks. 920 921.... 922 'checkpoint' LF 923 LF? 924.... 925 926Note that fast-import automatically switches packfiles when the current 927packfile reaches --max-pack-size, or 4 GiB, whichever limit is 928smaller. During an automatic packfile switch fast-import does not update 929the branch refs, tags or marks. 930 931As a `checkpoint` can require a significant amount of CPU time and 932disk IO (to compute the overall pack SHA-1 checksum, generate the 933corresponding index file, and update the refs) it can easily take 934several minutes for a single `checkpoint` command to complete. 935 936Frontends may choose to issue checkpoints during extremely large 937and long running imports, or when they need to allow another Git 938process access to a branch. However given that a 30 GiB Subversion 939repository can be loaded into Git through fast-import in about 3 hours, 940explicit checkpointing may not be necessary. 941 942The `LF` after the command is optional (it used to be required). 943 944`progress` 945~~~~~~~~~~ 946Causes fast-import to print the entire `progress` line unmodified to 947its standard output channel (file descriptor 1) when the command is 948processed from the input stream. The command otherwise has no impact 949on the current import, or on any of fast-import's internal state. 950 951.... 952 'progress' SP <any> LF 953 LF? 954.... 955 956The `<any>` part of the command may contain any sequence of bytes 957that does not contain `LF`. The `LF` after the command is optional. 958Callers may wish to process the output through a tool such as sed to 959remove the leading part of the line, for example: 960 961==== 962 frontend | git fast-import | sed 's/^progress //' 963==== 964 965Placing a `progress` command immediately after a `checkpoint` will 966inform the reader when the `checkpoint` has been completed and it 967can safely access the refs that fast-import updated. 968 969`get-mark` 970~~~~~~~~~~ 971Causes fast-import to print the SHA-1 corresponding to a mark to 972stdout or to the file descriptor previously arranged with the 973`--cat-blob-fd` argument. The command otherwise has no impact on the 974current import; its purpose is to retrieve SHA-1s that later commits 975might want to refer to in their commit messages. 976 977.... 978 'get-mark' SP ':' <idnum> LF 979.... 980 981See ``Responses To Commands'' below for details about how to read 982this output safely. 983 984`cat-blob` 985~~~~~~~~~~ 986Causes fast-import to print a blob to a file descriptor previously 987arranged with the `--cat-blob-fd` argument. The command otherwise 988has no impact on the current import; its main purpose is to 989retrieve blobs that may be in fast-import's memory but not 990accessible from the target repository. 991 992.... 993 'cat-blob' SP <dataref> LF 994.... 995 996The `<dataref>` can be either a mark reference (`:<idnum>`) 997set previously or a full 40-byte SHA-1 of a Git blob, preexisting or 998ready to be written. 9991000Output uses the same format as `git cat-file --batch`:10011002====1003 <sha1> SP 'blob' SP <size> LF1004 <contents> LF1005====10061007This command can be used where a `filemodify` directive can appear,1008allowing it to be used in the middle of a commit. For a `filemodify`1009using an inline directive, it can also appear right before the `data`1010directive.10111012See ``Responses To Commands'' below for details about how to read1013this output safely.10141015`ls`1016~~~~1017Prints information about the object at a path to a file descriptor1018previously arranged with the `--cat-blob-fd` argument. This allows1019printing a blob from the active commit (with `cat-blob`) or copying a1020blob or tree from a previous commit for use in the current one (with1021`filemodify`).10221023The `ls` command can also be used where a `filemodify` directive can1024appear, allowing it to be used in the middle of a commit.10251026Reading from the active commit::1027 This form can only be used in the middle of a `commit`.1028 The path names a directory entry within fast-import's1029 active commit. The path must be quoted in this case.1030+1031....1032 'ls' SP <path> LF1033....10341035Reading from a named tree::1036 The `<dataref>` can be a mark reference (`:<idnum>`) or the1037 full 40-byte SHA-1 of a Git tag, commit, or tree object,1038 preexisting or waiting to be written.1039 The path is relative to the top level of the tree1040 named by `<dataref>`.1041+1042....1043 'ls' SP <dataref> SP <path> LF1044....10451046See `filemodify` above for a detailed description of `<path>`.10471048Output uses the same format as `git ls-tree <tree> -- <path>`:10491050====1051 <mode> SP ('blob' | 'tree' | 'commit') SP <dataref> HT <path> LF1052====10531054The <dataref> represents the blob, tree, or commit object at <path>1055and can be used in later 'get-mark', 'cat-blob', 'filemodify', or1056'ls' commands.10571058If there is no file or subtree at that path, 'git fast-import' will1059instead report10601061====1062 missing SP <path> LF1063====10641065See ``Responses To Commands'' below for details about how to read1066this output safely.10671068`feature`1069~~~~~~~~~1070Require that fast-import supports the specified feature, or abort if1071it does not.10721073....1074 'feature' SP <feature> ('=' <argument>)? LF1075....10761077The <feature> part of the command may be any one of the following:10781079date-format::1080export-marks::1081relative-marks::1082no-relative-marks::1083force::1084 Act as though the corresponding command-line option with1085 a leading `--` was passed on the command line1086 (see OPTIONS, above).10871088import-marks::1089import-marks-if-exists::1090 Like --import-marks except in two respects: first, only one1091 "feature import-marks" or "feature import-marks-if-exists"1092 command is allowed per stream; second, an --import-marks=1093 or --import-marks-if-exists command-line option overrides1094 any of these "feature" commands in the stream; third,1095 "feature import-marks-if-exists" like a corresponding1096 command-line option silently skips a nonexistent file.10971098get-mark::1099cat-blob::1100ls::1101 Require that the backend support the 'get-mark', 'cat-blob',1102 or 'ls' command respectively.1103 Versions of fast-import not supporting the specified command1104 will exit with a message indicating so.1105 This lets the import error out early with a clear message,1106 rather than wasting time on the early part of an import1107 before the unsupported command is detected.11081109notes::1110 Require that the backend support the 'notemodify' (N)1111 subcommand to the 'commit' command.1112 Versions of fast-import not supporting notes will exit1113 with a message indicating so.11141115done::1116 Error out if the stream ends without a 'done' command.1117 Without this feature, errors causing the frontend to end1118 abruptly at a convenient point in the stream can go1119 undetected. This may occur, for example, if an import1120 front end dies in mid-operation without emitting SIGTERM1121 or SIGKILL at its subordinate git fast-import instance.11221123`option`1124~~~~~~~~1125Processes the specified option so that git fast-import behaves in a1126way that suits the frontend's needs.1127Note that options specified by the frontend are overridden by any1128options the user may specify to git fast-import itself.11291130....1131 'option' SP <option> LF1132....11331134The `<option>` part of the command may contain any of the options1135listed in the OPTIONS section that do not change import semantics,1136without the leading `--` and is treated in the same way.11371138Option commands must be the first commands on the input (not counting1139feature commands), to give an option command after any non-option1140command is an error.11411142The following command-line options change import semantics and may therefore1143not be passed as option:11441145* date-format1146* import-marks1147* export-marks1148* cat-blob-fd1149* force11501151`done`1152~~~~~~1153If the `done` feature is not in use, treated as if EOF was read.1154This can be used to tell fast-import to finish early.11551156If the `--done` command-line option or `feature done` command is1157in use, the `done` command is mandatory and marks the end of the1158stream.11591160RESPONSES TO COMMANDS1161---------------------1162New objects written by fast-import are not available immediately.1163Most fast-import commands have no visible effect until the next1164checkpoint (or completion). The frontend can send commands to1165fill fast-import's input pipe without worrying about how quickly1166they will take effect, which improves performance by simplifying1167scheduling.11681169For some frontends, though, it is useful to be able to read back1170data from the current repository as it is being updated (for1171example when the source material describes objects in terms of1172patches to be applied to previously imported objects). This can1173be accomplished by connecting the frontend and fast-import via1174bidirectional pipes:11751176====1177 mkfifo fast-import-output1178 frontend <fast-import-output |1179 git fast-import >fast-import-output1180====11811182A frontend set up this way can use `progress`, `get-mark`, `ls`, and1183`cat-blob` commands to read information from the import in progress.11841185To avoid deadlock, such frontends must completely consume any1186pending output from `progress`, `ls`, `get-mark`, and `cat-blob` before1187performing writes to fast-import that might block.11881189CRASH REPORTS1190-------------1191If fast-import is supplied invalid input it will terminate with a1192non-zero exit status and create a crash report in the top level of1193the Git repository it was importing into. Crash reports contain1194a snapshot of the internal fast-import state as well as the most1195recent commands that lead up to the crash.11961197All recent commands (including stream comments, file changes and1198progress commands) are shown in the command history within the crash1199report, but raw file data and commit messages are excluded from the1200crash report. This exclusion saves space within the report file1201and reduces the amount of buffering that fast-import must perform1202during execution.12031204After writing a crash report fast-import will close the current1205packfile and export the marks table. This allows the frontend1206developer to inspect the repository state and resume the import from1207the point where it crashed. The modified branches and tags are not1208updated during a crash, as the import did not complete successfully.1209Branch and tag information can be found in the crash report and1210must be applied manually if the update is needed.12111212An example crash:12131214====1215 $ cat >in <<END_OF_INPUT1216 # my very first test commit1217 commit refs/heads/master1218 committer Shawn O. Pearce <spearce> 19283 -04001219 # who is that guy anyway?1220 data <<EOF1221 this is my commit1222 EOF1223 M 644 inline .gitignore1224 data <<EOF1225 .gitignore1226 EOF1227 M 777 inline bob1228 END_OF_INPUT12291230 $ git fast-import <in1231 fatal: Corrupt mode: M 777 inline bob1232 fast-import: dumping crash report to .git/fast_import_crash_843412331234 $ cat .git/fast_import_crash_84341235 fast-import crash report:1236 fast-import process: 84341237 parent process : 13911238 at Sat Sep 1 00:58:12 200712391240 fatal: Corrupt mode: M 777 inline bob12411242 Most Recent Commands Before Crash1243 ---------------------------------1244 # my very first test commit1245 commit refs/heads/master1246 committer Shawn O. Pearce <spearce> 19283 -04001247 # who is that guy anyway?1248 data <<EOF1249 M 644 inline .gitignore1250 data <<EOF1251 * M 777 inline bob12521253 Active Branch LRU1254 -----------------1255 active_branches = 1 cur, 5 max12561257 pos clock name1258 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1259 1) 0 refs/heads/master12601261 Inactive Branches1262 -----------------1263 refs/heads/master:1264 status : active loaded dirty1265 tip commit : 00000000000000000000000000000000000000001266 old tree : 00000000000000000000000000000000000000001267 cur tree : 00000000000000000000000000000000000000001268 commit clock: 01269 last pack :127012711272 -------------------1273 END OF CRASH REPORT1274====12751276TIPS AND TRICKS1277---------------1278The following tips and tricks have been collected from various1279users of fast-import, and are offered here as suggestions.12801281Use One Mark Per Commit1282~~~~~~~~~~~~~~~~~~~~~~~1283When doing a repository conversion, use a unique mark per commit1284(`mark :<n>`) and supply the --export-marks option on the command1285line. fast-import will dump a file which lists every mark and the Git1286object SHA-1 that corresponds to it. If the frontend can tie1287the marks back to the source repository, it is easy to verify the1288accuracy and completeness of the import by comparing each Git1289commit to the corresponding source revision.12901291Coming from a system such as Perforce or Subversion this should be1292quite simple, as the fast-import mark can also be the Perforce changeset1293number or the Subversion revision number.12941295Freely Skip Around Branches1296~~~~~~~~~~~~~~~~~~~~~~~~~~~1297Don't bother trying to optimize the frontend to stick to one branch1298at a time during an import. Although doing so might be slightly1299faster for fast-import, it tends to increase the complexity of the frontend1300code considerably.13011302The branch LRU builtin to fast-import tends to behave very well, and the1303cost of activating an inactive branch is so low that bouncing around1304between branches has virtually no impact on import performance.13051306Handling Renames1307~~~~~~~~~~~~~~~~1308When importing a renamed file or directory, simply delete the old1309name(s) and modify the new name(s) during the corresponding commit.1310Git performs rename detection after-the-fact, rather than explicitly1311during a commit.13121313Use Tag Fixup Branches1314~~~~~~~~~~~~~~~~~~~~~~1315Some other SCM systems let the user create a tag from multiple1316files which are not from the same commit/changeset. Or to create1317tags which are a subset of the files available in the repository.13181319Importing these tags as-is in Git is impossible without making at1320least one commit which ``fixes up'' the files to match the content1321of the tag. Use fast-import's `reset` command to reset a dummy branch1322outside of your normal branch space to the base commit for the tag,1323then commit one or more file fixup commits, and finally tag the1324dummy branch.13251326For example since all normal branches are stored under `refs/heads/`1327name the tag fixup branch `TAG_FIXUP`. This way it is impossible for1328the fixup branch used by the importer to have namespace conflicts1329with real branches imported from the source (the name `TAG_FIXUP`1330is not `refs/heads/TAG_FIXUP`).13311332When committing fixups, consider using `merge` to connect the1333commit(s) which are supplying file revisions to the fixup branch.1334Doing so will allow tools such as 'git blame' to track1335through the real commit history and properly annotate the source1336files.13371338After fast-import terminates the frontend will need to do `rm .git/TAG_FIXUP`1339to remove the dummy branch.13401341Import Now, Repack Later1342~~~~~~~~~~~~~~~~~~~~~~~~1343As soon as fast-import completes the Git repository is completely valid1344and ready for use. Typically this takes only a very short time,1345even for considerably large projects (100,000+ commits).13461347However repacking the repository is necessary to improve data1348locality and access performance. It can also take hours on extremely1349large projects (especially if -f and a large --window parameter is1350used). Since repacking is safe to run alongside readers and writers,1351run the repack in the background and let it finish when it finishes.1352There is no reason to wait to explore your new Git project!13531354If you choose to wait for the repack, don't try to run benchmarks1355or performance tests until repacking is completed. fast-import outputs1356suboptimal packfiles that are simply never seen in real use1357situations.13581359Repacking Historical Data1360~~~~~~~~~~~~~~~~~~~~~~~~~1361If you are repacking very old imported data (e.g. older than the1362last year), consider expending some extra CPU time and supplying1363--window=50 (or higher) when you run 'git repack'.1364This will take longer, but will also produce a smaller packfile.1365You only need to expend the effort once, and everyone using your1366project will benefit from the smaller repository.13671368Include Some Progress Messages1369~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1370Every once in a while have your frontend emit a `progress` message1371to fast-import. The contents of the messages are entirely free-form,1372so one suggestion would be to output the current month and year1373each time the current commit date moves into the next month.1374Your users will feel better knowing how much of the data stream1375has been processed.137613771378PACKFILE OPTIMIZATION1379---------------------1380When packing a blob fast-import always attempts to deltify against the last1381blob written. Unless specifically arranged for by the frontend,1382this will probably not be a prior version of the same file, so the1383generated delta will not be the smallest possible. The resulting1384packfile will be compressed, but will not be optimal.13851386Frontends which have efficient access to all revisions of a1387single file (for example reading an RCS/CVS ,v file) can choose1388to supply all revisions of that file as a sequence of consecutive1389`blob` commands. This allows fast-import to deltify the different file1390revisions against each other, saving space in the final packfile.1391Marks can be used to later identify individual file revisions during1392a sequence of `commit` commands.13931394The packfile(s) created by fast-import do not encourage good disk access1395patterns. This is caused by fast-import writing the data in the order1396it is received on standard input, while Git typically organizes1397data within packfiles to make the most recent (current tip) data1398appear before historical data. Git also clusters commits together,1399speeding up revision traversal through better cache locality.14001401For this reason it is strongly recommended that users repack the1402repository with `git repack -a -d` after fast-import completes, allowing1403Git to reorganize the packfiles for faster data access. If blob1404deltas are suboptimal (see above) then also adding the `-f` option1405to force recomputation of all deltas can significantly reduce the1406final packfile size (30-50% smaller can be quite typical).14071408Instead of running `git repack` you can also run `git gc1409--aggressive`, which will also optimize other things after an import1410(e.g. pack loose refs). As noted in the "AGGRESSIVE" section in1411linkgit:git-gc[1] the `--aggressive` option will find new deltas with1412the `-f` option to linkgit:git-repack[1]. For the reasons elaborated1413on above using `--aggressive` after a fast-import is one of the few1414cases where it's known to be worthwhile.14151416MEMORY UTILIZATION1417------------------1418There are a number of factors which affect how much memory fast-import1419requires to perform an import. Like critical sections of core1420Git, fast-import uses its own memory allocators to amortize any overheads1421associated with malloc. In practice fast-import tends to amortize any1422malloc overheads to 0, due to its use of large block allocations.14231424per object1425~~~~~~~~~~1426fast-import maintains an in-memory structure for every object written in1427this execution. On a 32 bit system the structure is 32 bytes,1428on a 64 bit system the structure is 40 bytes (due to the larger1429pointer sizes). Objects in the table are not deallocated until1430fast-import terminates. Importing 2 million objects on a 32 bit system1431will require approximately 64 MiB of memory.14321433The object table is actually a hashtable keyed on the object name1434(the unique SHA-1). This storage configuration allows fast-import to reuse1435an existing or already written object and avoid writing duplicates1436to the output packfile. Duplicate blobs are surprisingly common1437in an import, typically due to branch merges in the source.14381439per mark1440~~~~~~~~1441Marks are stored in a sparse array, using 1 pointer (4 bytes or 81442bytes, depending on pointer size) per mark. Although the array1443is sparse, frontends are still strongly encouraged to use marks1444between 1 and n, where n is the total number of marks required for1445this import.14461447per branch1448~~~~~~~~~~1449Branches are classified as active and inactive. The memory usage1450of the two classes is significantly different.14511452Inactive branches are stored in a structure which uses 96 or 1201453bytes (32 bit or 64 bit systems, respectively), plus the length of1454the branch name (typically under 200 bytes), per branch. fast-import will1455easily handle as many as 10,000 inactive branches in under 2 MiB1456of memory.14571458Active branches have the same overhead as inactive branches, but1459also contain copies of every tree that has been recently modified on1460that branch. If subtree `include` has not been modified since the1461branch became active, its contents will not be loaded into memory,1462but if subtree `src` has been modified by a commit since the branch1463became active, then its contents will be loaded in memory.14641465As active branches store metadata about the files contained on that1466branch, their in-memory storage size can grow to a considerable size1467(see below).14681469fast-import automatically moves active branches to inactive status based on1470a simple least-recently-used algorithm. The LRU chain is updated on1471each `commit` command. The maximum number of active branches can be1472increased or decreased on the command line with --active-branches=.14731474per active tree1475~~~~~~~~~~~~~~~1476Trees (aka directories) use just 12 bytes of memory on top of the1477memory required for their entries (see ``per active file'' below).1478The cost of a tree is virtually 0, as its overhead amortizes out1479over the individual file entries.14801481per active file entry1482~~~~~~~~~~~~~~~~~~~~~1483Files (and pointers to subtrees) within active trees require 52 or 641484bytes (32/64 bit platforms) per entry. To conserve space, file and1485tree names are pooled in a common string table, allowing the filename1486``Makefile'' to use just 16 bytes (after including the string header1487overhead) no matter how many times it occurs within the project.14881489The active branch LRU, when coupled with the filename string pool1490and lazy loading of subtrees, allows fast-import to efficiently import1491projects with 2,000+ branches and 45,114+ files in a very limited1492memory footprint (less than 2.7 MiB per active branch).14931494SIGNALS1495-------1496Sending *SIGUSR1* to the 'git fast-import' process ends the current1497packfile early, simulating a `checkpoint` command. The impatient1498operator can use this facility to peek at the objects and refs from an1499import in progress, at the cost of some added running time and worse1500compression.15011502SEE ALSO1503--------1504linkgit:git-fast-export[1]15051506GIT1507---1508Part of the linkgit:git[1] suite