1git-fast-import(1) 2================== 3 4NAME 5---- 6git-fast-import - Backend for fast Git data importers. 7 8 9SYNOPSIS 10-------- 11frontend | 'git-fast-import' [options] 12 13DESCRIPTION 14----------- 15This program is usually not what the end user wants to run directly. 16Most end users want to use one of the existing frontend programs, 17which parses a specific type of foreign source and feeds the contents 18stored there to git-fast-import (gfi). 19 20gfi reads a mixed command/data stream from standard input and 21writes one or more packfiles directly into the current repository. 22When EOF is received on standard input, fast import writes out 23updated branch and tag refs, fully updating the current repository 24with the newly imported data. 25 26The gfi backend itself can import into an empty repository (one that 27has already been initialized by gitlink:git-init[1]) or incrementally 28update an existing populated repository. Whether or not incremental 29imports are supported from a particular foreign source depends on 30the frontend program in use. 31 32 33OPTIONS 34------- 35--max-pack-size=<n>:: 36 Maximum size of each output packfile, expressed in MiB. 37 The default is 4096 (4 GiB) as that is the maximum allowed 38 packfile size (due to file format limitations). Some 39 importers may wish to lower this, such as to ensure the 40 resulting packfiles fit on CDs. 41 42--depth=<n>:: 43 Maximum delta depth, for blob and tree deltification. 44 Default is 10. 45 46--active-branches=<n>:: 47 Maximum number of branches to maintain active at once. 48 See ``Memory Utilization'' below for details. Default is 5. 49 50--export-marks=<file>:: 51 Dumps the internal marks table to <file> when complete. 52 Marks are written one per line as `:markid SHA-1`. 53 Frontends can use this file to validate imports after they 54 have been completed. 55 56 57Performance 58----------- 59The design of gfi allows it to import large projects in a minimum 60amount of memory usage and processing time. Assuming the frontend 61is able to keep up with gfi and feed it a constant stream of data, 62import times for projects holding 10+ years of history and containing 63100,000+ individual commits are generally completed in just 1-2 64hours on quite modest (~$2,000 USD) hardware. 65 66Most bottlenecks appear to be in foreign source data access (the 67source just cannot extract revisions fast enough) or disk IO (gfi 68writes as fast as the disk will take the data). Imports will run 69faster if the source data is stored on a different drive than the 70destination Git repository (due to less IO contention). 71 72 73Development Cost 74---------------- 75A typical frontend for gfi tends to weigh in at approximately 200 76lines of Perl/Python/Ruby code. Most developers have been able to 77create working importers in just a couple of hours, even though it 78is their first exposure to gfi, and sometimes even to Git. This is 79an ideal situation, given that most conversion tools are throw-away 80(use once, and never look back). 81 82 83Parallel Operation 84------------------ 85Like `git-push` or `git-fetch`, imports handled by gfi are safe to 86run alongside parallel `git repack -a -d` or `git gc` invocations, 87or any other Git operation (including `git prune`, as loose objects 88are never used by gfi). 89 90However, gfi does not lock the branch or tag refs it is actively 91importing. After EOF, during its ref update phase, gfi blindly 92overwrites each imported branch or tag ref. Consequently it is not 93safe to modify refs that are currently being used by a running gfi 94instance, as work could be lost when gfi overwrites the refs. 95 96 97Technical Discussion 98-------------------- 99gfi tracks a set of branches in memory. Any branch can be created 100or modified at any point during the import process by sending a 101`commit` command on the input stream. This design allows a frontend 102program to process an unlimited number of branches simultaneously, 103generating commits in the order they are available from the source 104data. It also simplifies the frontend programs considerably. 105 106gfi does not use or alter the current working directory, or any 107file within it. (It does however update the current Git repository, 108as referenced by `GIT_DIR`.) Therefore an import frontend may use 109the working directory for its own purposes, such as extracting file 110revisions from the foreign source. This ignorance of the working 111directory also allows gfi to run very quickly, as it does not 112need to perform any costly file update operations when switching 113between branches. 114 115Input Format 116------------ 117With the exception of raw file data (which Git does not interpret) 118the gfi input format is text (ASCII) based. This text based 119format simplifies development and debugging of frontend programs, 120especially when a higher level language such as Perl, Python or 121Ruby is being used. 122 123gfi is very strict about its input. Where we say SP below we mean 124*exactly* one space. Likewise LF means one (and only one) linefeed. 125Supplying additional whitespace characters will cause unexpected 126results, such as branch names or file names with leading or trailing 127spaces in their name, or early termination of gfi when it encounters 128unexpected input. 129 130Commands 131~~~~~~~~ 132gfi accepts several commands to update the current repository 133and control the current import process. More detailed discussion 134(with examples) of each command follows later. 135 136`commit`:: 137 Creates a new branch or updates an existing branch by 138 creating a new commit and updating the branch to point at 139 the newly created commit. 140 141`tag`:: 142 Creates an annotated tag object from an existing commit or 143 branch. Lightweight tags are not supported by this command, 144 as they are not recommended for recording meaningful points 145 in time. 146 147`reset`:: 148 Reset an existing branch (or a new branch) to a specific 149 revision. This command must be used to change a branch to 150 a specific revision without making a commit on it. 151 152`blob`:: 153 Convert raw file data into a blob, for future use in a 154 `commit` command. This command is optional and is not 155 needed to perform an import. 156 157`checkpoint`:: 158 Forces gfi to close the current packfile, generate its 159 unique SHA-1 checksum and index, and start a new packfile. 160 This command is optional and is not needed to perform 161 an import. 162 163`commit` 164~~~~~~~~ 165Create or update a branch with a new commit, recording one logical 166change to the project. 167 168.... 169 'commit' SP <ref> LF 170 mark? 171 ('author' SP <name> SP LT <email> GT SP <time> SP <tz> LF)? 172 'committer' SP <name> SP LT <email> GT SP <time> SP <tz> LF 173 data 174 ('from' SP <committish> LF)? 175 ('merge' SP <committish> LF)? 176 (filemodify | filedelete)* 177 LF 178.... 179 180where `<ref>` is the name of the branch to make the commit on. 181Typically branch names are prefixed with `refs/heads/` in 182Git, so importing the CVS branch symbol `RELENG-1_0` would use 183`refs/heads/RELENG-1_0` for the value of `<ref>`. The value of 184`<ref>` must be a valid refname in Git. As `LF` is not valid in 185a Git refname, no quoting or escaping syntax is supported here. 186 187A `mark` command may optionally appear, requesting gfi to save a 188reference to the newly created commit for future use by the frontend 189(see below for format). It is very common for frontends to mark 190every commit they create, thereby allowing future branch creation 191from any imported commit. 192 193The `data` command following `committer` must supply the commit 194message (see below for `data` command syntax). To import an empty 195commit message use a 0 length data. Commit messages are free-form 196and are not interpreted by Git. Currently they must be encoded in 197UTF-8, as gfi does not permit other encodings to be specified. 198 199Zero or more `filemodify` and `filedelete` commands may be 200included to update the contents of the branch prior to the commit. 201These commands can be supplied in any order, gfi is not sensitive 202to pathname or operation ordering. 203 204`author` 205^^^^^^^^ 206An `author` command may optionally appear, if the author information 207might differ from the committer information. If `author` is omitted 208then gfi will automatically use the committer's information for 209the author portion of the commit. See below for a description of 210the fields in `author`, as they are identical to `committer`. 211 212`committer` 213^^^^^^^^^^^ 214The `committer` command indicates who made this commit, and when 215they made it. 216 217Here `<name>` is the person's display name (for example 218``Com M Itter'') and `<email>` is the person's email address 219(``cm@example.com''). `LT` and `GT` are the literal less-than (\x3c) 220and greater-than (\x3e) symbols. These are required to delimit 221the email address from the other fields in the line. Note that 222`<name>` is free-form and may contain any sequence of bytes, except 223`LT` and `LF`. It is typically UTF-8 encoded. 224 225The time of the change is specified by `<time>` as the number of 226seconds since the UNIX epoc (midnight, Jan 1, 1970, UTC) and is 227written in base-10 notation using US-ASCII digits. The committer's 228timezone is specified by `<tz>` as a positive or negative offset 229from UTC. For example EST (which is typically 5 hours behind GMT) 230would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''. 231 232`from` 233^^^^^^ 234Only valid for the first commit made on this branch by this 235gfi process. The `from` command is used to specify the commit 236to initialize this branch from. This revision will be the first 237ancestor of the new commit. 238 239Omitting the `from` command in the first commit of a new branch will 240cause gfi to create that commit with no ancestor. This tends to be 241desired only for the initial commit of a project. Omitting the 242`from` command on existing branches is required, as the current 243commit on that branch is automatically assumed to be the first 244ancestor of the new commit. 245 246As `LF` is not valid in a Git refname or SHA-1 expression, no 247quoting or escaping syntax is supported within `<committish>`. 248 249Here `<committish>` is any of the following: 250 251* The name of an existing branch already in gfi's internal branch 252 table. If gfi doesn't know the name, its treated as a SHA-1 253 expression. 254 255* A mark reference, `:<idnum>`, where `<idnum>` is the mark number. 256+ 257The reason gfi uses `:` to denote a mark reference is this character 258is not legal in a Git branch name. The leading `:` makes it easy 259to distingush between the mark 42 (`:42`) and the branch 42 (`42` 260or `refs/heads/42`), or an abbreviated SHA-1 which happened to 261consist only of base-10 digits. 262+ 263Marks must be declared (via `mark`) before they can be used. 264 265* A complete 40 byte or abbreviated commit SHA-1 in hex. 266 267* Any valid Git SHA-1 expression that resolves to a commit. See 268 ``SPECIFYING REVISIONS'' in gitlink:git-rev-parse[1] for details. 269 270The special case of restarting an incremental import from the 271current branch value should be written as: 272---- 273 from refs/heads/branch^0 274---- 275The `^0` suffix is necessary as gfi does not permit a branch to 276start from itself, and the branch is created in memory before the 277`from` command is even read from the input. Adding `^0` will force 278gfi to resolve the commit through Git's revision parsing library, 279rather than its internal branch table, thereby loading in the 280existing value of the branch. 281 282`merge` 283^^^^^^^ 284Includes one additional ancestor commit, and makes the current 285commit a merge commit. An unlimited number of `merge` commands per 286commit are permitted by gfi, thereby establishing an n-way merge. 287However Git's other tools never create commits with more than 15 288additional ancestors (forming a 16-way merge). For this reason 289it is suggested that frontends do not use more than 15 `merge` 290commands per commit. 291 292Here `<committish>` is any of the commit specification expressions 293also accepted by `from` (see above). 294 295`filemodify` 296^^^^^^^^^^ 297Included in a `commit` command to add a new file or change the 298content of an existing file. This command has two different means 299of specifying the content of the file. 300 301External data format:: 302 The data content for the file was already supplied by a prior 303 `blob` command. The frontend just needs to connect it. 304+ 305.... 306 'M' SP <mode> SP <dataref> SP <path> LF 307.... 308+ 309Here `<dataref>` can be either a mark reference (`:<idnum>`) 310set by a prior `blob` command, or a full 40-byte SHA-1 of an 311existing Git blob object. 312 313Inline data format:: 314 The data content for the file has not been supplied yet. 315 The frontend wants to supply it as part of this modify 316 command. 317+ 318.... 319 'M' SP <mode> SP 'inline' SP <path> LF 320 data 321.... 322+ 323See below for a detailed description of the `data` command. 324 325In both formats `<mode>` is the type of file entry, specified 326in octal. Git only supports the following modes: 327 328* `100644` or `644`: A normal (not-executable) file. The majority 329 of files in most projects use this mode. If in doubt, this is 330 what you want. 331* `100755` or `755`: A normal, but executable, file. 332* `140000`: A symlink, the content of the file will be the link target. 333 334In both formats `<path>` is the complete path of the file to be added 335(if not already existing) or modified (if already existing). 336 337A `<path>` string must use UNIX-style directory seperators (forward 338slash `/`), may contain any byte other than `LF`, and must not 339start with double quote (`"`). 340 341If an `LF` or double quote must be encoded into `<path>` shell-style 342quoting should be used, e.g. `"path/with\n and \" in it"`. 343 344The value of `<path>` must be in canoncial form. That is it must not: 345 346* contain an empty directory component (e.g. `foo//bar` is invalid), 347* end with a directory seperator (e.g. `foo/` is invalid), 348* start with a directory seperator (e.g. `/foo` is invalid), 349* contain the special component `.` or `..` (e.g. `foo/./bar` and 350 `foo/../bar` are invalid). 351 352It is recommended that `<path>` always be encoded using UTF-8. 353 354 355`filedelete` 356^^^^^^^^^^ 357Included in a `commit` command to remove a file from the branch. 358If the file removal makes its directory empty, the directory will 359be automatically removed too. This cascades up the tree until the 360first non-empty directory or the root is reached. 361 362.... 363 'D' SP <path> LF 364.... 365 366here `<path>` is the complete path of the file to be removed. 367See `filemodify` above for a detailed description of `<path>`. 368 369`mark` 370~~~~~~ 371Arranges for gfi to save a reference to the current object, allowing 372the frontend to recall this object at a future point in time, without 373knowing its SHA-1. Here the current object is the object creation 374command the `mark` command appears within. This can be `commit`, 375`tag`, and `blob`, but `commit` is the most common usage. 376 377.... 378 'mark' SP ':' <idnum> LF 379.... 380 381where `<idnum>` is the number assigned by the frontend to this mark. 382The value of `<idnum>` is expressed in base 10 notation using 383US-ASCII digits. The value 0 is reserved and cannot be used as 384a mark. Only values greater than or equal to 1 may be used as marks. 385 386New marks are created automatically. Existing marks can be moved 387to another object simply by reusing the same `<idnum>` in another 388`mark` command. 389 390`tag` 391~~~~~ 392Creates an annotated tag referring to a specific commit. To create 393lightweight (non-annotated) tags see the `reset` command below. 394 395.... 396 'tag' SP <name> LF 397 'from' SP <committish> LF 398 'tagger' SP <name> SP LT <email> GT SP <time> SP <tz> LF 399 data 400 LF 401.... 402 403where `<name>` is the name of the tag to create. 404 405Tag names are automatically prefixed with `refs/tags/` when stored 406in Git, so importing the CVS branch symbol `RELENG-1_0-FINAL` would 407use just `RELENG-1_0-FINAL` for `<name>`, and gfi will write the 408corresponding ref as `refs/tags/RELENG-1_0-FINAL`. 409 410The value of `<name>` must be a valid refname in Git and therefore 411may contain forward slashes. As `LF` is not valid in a Git refname, 412no quoting or escaping syntax is supported here. 413 414The `from` command is the same as in the `commit` command; see 415above for details. 416 417The `tagger` command uses the same format as `committer` within 418`commit`; again see above for details. 419 420The `data` command following `tagger` must supply the annotated tag 421message (see below for `data` command syntax). To import an empty 422tag message use a 0 length data. Tag messages are free-form and are 423not interpreted by Git. Currently they must be encoded in UTF-8, 424as gfi does not permit other encodings to be specified. 425 426Signing annotated tags during import from within gfi is not 427supported. Trying to include your own PGP/GPG signature is not 428recommended, as the frontend does not (easily) have access to the 429complete set of bytes which normally goes into such a signature. 430If signing is required, create lightweight tags from within gfi with 431`reset`, then create the annotated versions of those tags offline 432with the standard gitlink:git-tag[1] process. 433 434`reset` 435~~~~~~~ 436Creates (or recreates) the named branch, optionally starting from 437a specific revision. The reset command allows a frontend to issue 438a new `from` command for an existing branch, or to create a new 439branch from an existing commit without creating a new commit. 440 441.... 442 'reset' SP <ref> LF 443 ('from' SP <committish> LF)? 444 LF 445.... 446 447For a detailed description of `<ref>` and `<committish>` see above 448under `commit` and `from`. 449 450The `reset` command can also be used to create lightweight 451(non-annotated) tags. For example: 452 453==== 454 reset refs/tags/938 455 from :938 456==== 457 458would create the lightweight tag `refs/tags/938` referring to 459whatever commit mark `:938` references. 460 461`blob` 462~~~~~~ 463Requests writing one file revision to the packfile. The revision 464is not connected to any commit; this connection must be formed in 465a subsequent `commit` command by referencing the blob through an 466assigned mark. 467 468.... 469 'blob' LF 470 mark? 471 data 472.... 473 474The mark command is optional here as some frontends have chosen 475to generate the Git SHA-1 for the blob on their own, and feed that 476directly to `commit`. This is typically more work than its worth 477however, as marks are inexpensive to store and easy to use. 478 479`data` 480~~~~~~ 481Supplies raw data (for use as blob/file content, commit messages, or 482annotated tag messages) to gfi. Data can be supplied using an exact 483byte count or delimited with a terminating line. Real frontends 484intended for production-quality conversions should always use the 485exact byte count format, as it is more robust and performs better. 486The delimited format is intended primarily for testing gfi. 487 488Exact byte count format: 489 490.... 491 'data' SP <count> LF 492 <raw> LF 493.... 494 495where `<count>` is the exact number of bytes appearing within 496`<raw>`. The value of `<count>` is expressed in base 10 notation 497using US-ASCII digits. The `LF` on either side of `<raw>` is not 498included in `<count>` and will not be included in the imported data. 499 500Delimited format: 501 502.... 503 'data' SP '<<' <delim> LF 504 <raw> LF 505 <delim> LF 506.... 507 508where `<delim>` is the chosen delimiter string. The string `<delim>` 509must not appear on a line by itself within `<raw>`, as otherwise 510gfi will think the data ends earlier than it really does. The `LF` 511immediately trailing `<raw>` is part of `<raw>`. This is one of 512the limitations of the delimited format, it is impossible to supply 513a data chunk which does not have an LF as its last byte. 514 515`checkpoint` 516~~~~~~~~~~~~ 517Forces gfi to close the current packfile and start a new one. 518As this requires a significant amount of CPU time and disk IO 519(to compute the overall pack SHA-1 checksum and generate the 520corresponding index file) it can easily take several minutes for 521a single `checkpoint` command to complete. 522 523.... 524 'checkpoint' LF 525 LF 526.... 527 528Packfile Optimization 529--------------------- 530When packing a blob gfi always attempts to deltify against the last 531blob written. Unless specifically arranged for by the frontend, 532this will probably not be a prior version of the same file, so the 533generated delta will not be the smallest possible. The resulting 534packfile will be compressed, but will not be optimal. 535 536Frontends which have efficient access to all revisions of a 537single file (for example reading an RCS/CVS ,v file) can choose 538to supply all revisions of that file as a sequence of consecutive 539`blob` commands. This allows gfi to deltify the different file 540revisions against each other, saving space in the final packfile. 541Marks can be used to later identify individual file revisions during 542a sequence of `commit` commands. 543 544The packfile(s) created by gfi do not encourage good disk access 545patterns. This is caused by gfi writing the data in the order 546it is received on standard input, while Git typically organizes 547data within packfiles to make the most recent (current tip) data 548appear before historical data. Git also clusters commits together, 549speeding up revision traversal through better cache locality. 550 551For this reason it is strongly recommended that users repack the 552repository with `git repack -a -d` after gfi completes, allowing 553Git to reorganize the packfiles for faster data access. If blob 554deltas are suboptimal (see above) then also adding the `-f` option 555to force recomputation of all deltas can significantly reduce the 556final packfile size (30-50% smaller can be quite typical). 557 558Memory Utilization 559------------------ 560There are a number of factors which affect how much memory gfi 561requires to perform an import. Like critical sections of core 562Git, gfi uses its own memory allocators to ammortize any overheads 563associated with malloc. In practice gfi tends to ammoritize any 564malloc overheads to 0, due to its use of large block allocations. 565 566per object 567~~~~~~~~~~ 568gfi maintains an in-memory structure for every object written in 569this execution. On a 32 bit system the structure is 32 bytes, 570on a 64 bit system the structure is 40 bytes (due to the larger 571pointer sizes). Objects in the table are not deallocated until 572gfi terminates. Importing 2 million objects on a 32 bit system 573will require approximately 64 MiB of memory. 574 575The object table is actually a hashtable keyed on the object name 576(the unique SHA-1). This storage configuration allows gfi to reuse 577an existing or already written object and avoid writing duplicates 578to the output packfile. Duplicate blobs are surprisingly common 579in an import, typically due to branch merges in the source. 580 581per mark 582~~~~~~~~ 583Marks are stored in a sparse array, using 1 pointer (4 bytes or 8 584bytes, depending on pointer size) per mark. Although the array 585is sparse, frontends are still strongly encouraged to use marks 586between 1 and n, where n is the total number of marks required for 587this import. 588 589per branch 590~~~~~~~~~~ 591Branches are classified as active and inactive. The memory usage 592of the two classes is significantly different. 593 594Inactive branches are stored in a structure which uses 96 or 120 595bytes (32 bit or 64 bit systems, respectively), plus the length of 596the branch name (typically under 200 bytes), per branch. gfi will 597easily handle as many as 10,000 inactive branches in under 2 MiB 598of memory. 599 600Active branches have the same overhead as inactive branches, but 601also contain copies of every tree that has been recently modified on 602that branch. If subtree `include` has not been modified since the 603branch became active, its contents will not be loaded into memory, 604but if subtree `src` has been modified by a commit since the branch 605became active, then its contents will be loaded in memory. 606 607As active branches store metadata about the files contained on that 608branch, their in-memory storage size can grow to a considerable size 609(see below). 610 611gfi automatically moves active branches to inactive status based on 612a simple least-recently-used algorithm. The LRU chain is updated on 613each `commit` command. The maximum number of active branches can be 614increased or decreased on the command line with `--active-branches=`. 615 616per active tree 617~~~~~~~~~~~~~~~ 618Trees (aka directories) use just 12 bytes of memory on top of the 619memory required for their entries (see ``per active file'' below). 620The cost of a tree is virtually 0, as its overhead ammortizes out 621over the individual file entries. 622 623per active file entry 624~~~~~~~~~~~~~~~~~~~~~ 625Files (and pointers to subtrees) within active trees require 52 or 64 626bytes (32/64 bit platforms) per entry. To conserve space, file and 627tree names are pooled in a common string table, allowing the filename 628``Makefile'' to use just 16 bytes (after including the string header 629overhead) no matter how many times it occurs within the project. 630 631The active branch LRU, when coupled with the filename string pool 632and lazy loading of subtrees, allows gfi to efficiently import 633projects with 2,000+ branches and 45,114+ files in a very limited 634memory footprint (less than 2.7 MiB per active branch). 635 636 637Author 638------ 639Written by Shawn O. Pearce <spearce@spearce.org>. 640 641Documentation 642-------------- 643Documentation by Shawn O. Pearce <spearce@spearce.org>. 644 645GIT 646--- 647Part of the gitlink:git[7] suite 648