1Core GIT Tests 2============== 3 4This directory holds many test scripts for core GIT tools. The 5first part of this short document describes how to run the tests 6and read their output. 7 8When fixing the tools or adding enhancements, you are strongly 9encouraged to add tests in this directory to cover what you are 10trying to fix or enhance. The later part of this short document 11describes how your test scripts should be organized. 12 13 14Running Tests 15------------- 16 17The easiest way to run tests is to say "make". This runs all 18the tests. 19 20 *** t0000-basic.sh *** 21 ok 1 - .git/objects should be empty after git init in an empty repo. 22 ok 2 - .git/objects should have 3 subdirectories. 23 ok 3 - success is reported like this 24 ... 25 ok 43 - very long name in the index handled sanely 26 # fixed 1 known breakage(s) 27 # still have 1 known breakage(s) 28 # passed all remaining 42 test(s) 29 1..43 30 *** t0001-init.sh *** 31 ok 1 - plain 32 ok 2 - plain with GIT_WORK_TREE 33 ok 3 - plain bare 34 35Since the tests all output TAP (see http://testanything.org) they can 36be run with any TAP harness. Here's an example of parallel testing 37powered by a recent version of prove(1): 38 39 $ prove --timer --jobs 15 ./t[0-9]*.sh 40 [19:17:33] ./t0005-signals.sh ................................... ok 36 ms 41 [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms 42 [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms 43 [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms 44 [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms 45 ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )=== 46 47prove and other harnesses come with a lot of useful options. The 48--state option in particular is very useful: 49 50 # Repeat until no more failures 51 $ prove -j 15 --state=failed,save ./t[0-9]*.sh 52 53You can also run each test individually from command line, like this: 54 55 $ sh ./t3010-ls-files-killed-modified.sh 56 ok 1 - git update-index --add to add various paths. 57 ok 2 - git ls-files -k to show killed files. 58 ok 3 - validate git ls-files -k output. 59 ok 4 - git ls-files -m to show modified files. 60 ok 5 - validate git ls-files -m output. 61 # passed all 5 test(s) 62 1..5 63 64You can pass --verbose (or -v), --debug (or -d), and --immediate 65(or -i) command line argument to the test, or by setting GIT_TEST_OPTS 66appropriately before running "make". 67 68--verbose:: 69 This makes the test more verbose. Specifically, the 70 command being run and their output if any are also 71 output. 72 73--debug:: 74 This may help the person who is developing a new test. 75 It causes the command defined with test_debug to run. 76 77--immediate:: 78 This causes the test to immediately exit upon the first 79 failed test. 80 81--long-tests:: 82 This causes additional long-running tests to be run (where 83 available), for more exhaustive testing. 84 85--valgrind:: 86 Execute all Git binaries with valgrind and exit with status 87 126 on errors (just like regular tests, this will only stop 88 the test script when running under -i). Valgrind errors 89 go to stderr, so you might want to pass the -v option, too. 90 91 Since it makes no sense to run the tests with --valgrind and 92 not see any output, this option implies --verbose. For 93 convenience, it also implies --tee. 94 95--tee:: 96 In addition to printing the test output to the terminal, 97 write it to files named 't/test-results/$TEST_NAME.out'. 98 As the names depend on the tests' file names, it is safe to 99 run the tests with this option in parallel. 100 101--with-dashes:: 102 By default tests are run without dashed forms of 103 commands (like git-commit) in the PATH (it only uses 104 wrappers from ../bin-wrappers). Use this option to include 105 the build directory (..) in the PATH, which contains all 106 the dashed forms of commands. This option is currently 107 implied by other options like --valgrind and 108 GIT_TEST_INSTALLED. 109 110--root=<directory>:: 111 Create "trash" directories used to store all temporary data during 112 testing under <directory>, instead of the t/ directory. 113 Using this option with a RAM-based filesystem (such as tmpfs) 114 can massively speed up the test suite. 115 116You can also set the GIT_TEST_INSTALLED environment variable to 117the bindir of an existing git installation to test that installation. 118You still need to have built this git sandbox, from which various 119test-* support programs, templates, and perl libraries are used. 120If your installed git is incomplete, it will silently test parts of 121your built version instead. 122 123When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to 124override the location of the dashed-form subcommands (what 125GIT_EXEC_PATH would be used for during normal operation). 126GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. 127 128 129Skipping Tests 130-------------- 131 132In some environments, certain tests have no way of succeeding 133due to platform limitation, such as lack of 'unzip' program, or 134filesystem that do not allow arbitrary sequence of non-NUL bytes 135as pathnames. 136 137You should be able to say something like 138 139 $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh 140 141and even: 142 143 $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make 144 145to omit such tests. The value of the environment variable is a 146SP separated list of patterns that tells which tests to skip, 147and either can match the "t[0-9]{4}" part to skip the whole 148test, or t[0-9]{4} followed by ".$number" to say which 149particular test to skip. 150 151Note that some tests in the existing test suite rely on previous 152test item, so you cannot arbitrarily disable one and expect the 153remainder of test to check what the test originally was intended 154to check. 155 156 157Naming Tests 158------------ 159 160The test files are named as: 161 162 tNNNN-commandname-details.sh 163 164where N is a decimal digit. 165 166First digit tells the family: 167 168 0 - the absolute basics and global stuff 169 1 - the basic commands concerning database 170 2 - the basic commands concerning the working tree 171 3 - the other basic commands (e.g. ls-files) 172 4 - the diff commands 173 5 - the pull and exporting commands 174 6 - the revision tree commands (even e.g. merge-base) 175 7 - the porcelainish commands concerning the working tree 176 8 - the porcelainish commands concerning forensics 177 9 - the git tools 178 179Second digit tells the particular command we are testing. 180 181Third digit (optionally) tells the particular switch or group of switches 182we are testing. 183 184If you create files under t/ directory (i.e. here) that is not 185the top-level test script, never name the file to match the above 186pattern. The Makefile here considers all such files as the 187top-level test script and tries to run all of them. A care is 188especially needed if you are creating a common test library 189file, similar to test-lib.sh, because such a library file may 190not be suitable for standalone execution. 191 192 193Writing Tests 194------------- 195 196The test script is written as a shell script. It should start 197with the standard "#!/bin/sh" with copyright notices, and an 198assignment to variable 'test_description', like this: 199 200 #!/bin/sh 201 # 202 # Copyright (c) 2005 Junio C Hamano 203 # 204 205 test_description='xxx test (option --frotz) 206 207 This test registers the following structure in the cache 208 and tries to run git-ls-files with option --frotz.' 209 210 211Source 'test-lib.sh' 212-------------------- 213 214After assigning test_description, the test script should source 215test-lib.sh like this: 216 217 . ./test-lib.sh 218 219This test harness library does the following things: 220 221 - If the script is invoked with command line argument --help 222 (or -h), it shows the test_description and exits. 223 224 - Creates an empty test directory with an empty .git/objects database 225 and chdir(2) into it. This directory is 't/trash 226 directory.$test_name_without_dotsh', with t/ subject to change by 227 the --root option documented above. 228 229 - Defines standard test helper functions for your scripts to 230 use. These functions are designed to make all scripts behave 231 consistently when command line arguments --verbose (or -v), 232 --debug (or -d), and --immediate (or -i) is given. 233 234Do's, don'ts & things to keep in mind 235------------------------------------- 236 237Here are a few examples of things you probably should and shouldn't do 238when writing tests. 239 240Do: 241 242 - Put all code inside test_expect_success and other assertions. 243 244 Even code that isn't a test per se, but merely some setup code 245 should be inside a test assertion. 246 247 - Chain your test assertions 248 249 Write test code like this: 250 251 git merge foo && 252 git push bar && 253 test ... 254 255 Instead of: 256 257 git merge hla 258 git push gh 259 test ... 260 261 That way all of the commands in your tests will succeed or fail. If 262 you must ignore the return value of something (e.g., the return 263 after unsetting a variable that was already unset is unportable) it's 264 best to indicate so explicitly with a semicolon: 265 266 unset HLAGH; 267 git merge hla && 268 git push gh && 269 test ... 270 271 - Check the test coverage for your tests. See the "Test coverage" 272 below. 273 274 Don't blindly follow test coverage metrics, they're a good way to 275 spot if you've missed something. If a new function you added 276 doesn't have any coverage you're probably doing something wrong, 277 but having 100% coverage doesn't necessarily mean that you tested 278 everything. 279 280 Tests that are likely to smoke out future regressions are better 281 than tests that just inflate the coverage metrics. 282 283Don't: 284 285 - exit() within a <script> part. 286 287 The harness will catch this as a programming error of the test. 288 Use test_done instead if you need to stop the tests early (see 289 "Skipping tests" below). 290 291 - Break the TAP output 292 293 The raw output from your test may be interpreted by a TAP harness. TAP 294 harnesses will ignore everything they don't know about, but don't step 295 on their toes in these areas: 296 297 - Don't print lines like "$x..$y" where $x and $y are integers. 298 299 - Don't print lines that begin with "ok" or "not ok". 300 301 TAP harnesses expect a line that begins with either "ok" and "not 302 ok" to signal a test passed or failed (and our harness already 303 produces such lines), so your script shouldn't emit such lines to 304 their output. 305 306 You can glean some further possible issues from the TAP grammar 307 (see http://search.cpan.org/perldoc?TAP::Parser::Grammar#TAP_Grammar) 308 but the best indication is to just run the tests with prove(1), 309 it'll complain if anything is amiss. 310 311Keep in mind: 312 313 - Inside <script> part, the standard output and standard error 314 streams are discarded, and the test harness only reports "ok" or 315 "not ok" to the end user running the tests. Under --verbose, they 316 are shown to help debugging the tests. 317 318 319Skipping tests 320-------------- 321 322If you need to skip tests you should do so be using the three-arg form 323of the test_* functions (see the "Test harness library" section 324below), e.g.: 325 326 test_expect_success PERL 'I need Perl' " 327 '$PERL_PATH' -e 'hlagh() if unf_unf()' 328 " 329 330The advantage of skipping tests like this is that platforms that don't 331have the PERL and other optional dependencies get an indication of how 332many tests they're missing. 333 334If the test code is too hairy for that (i.e. does a lot of setup work 335outside test assertions) you can also skip all remaining tests by 336setting skip_all and immediately call test_done: 337 338 if ! test_have_prereq PERL 339 then 340 skip_all='skipping perl interface tests, perl not available' 341 test_done 342 fi 343 344The string you give to skip_all will be used as an explanation for why 345the test was skipped. 346 347End with test_done 348------------------ 349 350Your script will be a sequence of tests, using helper functions 351from the test harness library. At the end of the script, call 352'test_done'. 353 354 355Test harness library 356-------------------- 357 358There are a handful helper functions defined in the test harness 359library for your script to use. 360 361 - test_expect_success [<prereq>] <message> <script> 362 363 Usually takes two strings as parameter, and evaluates the 364 <script>. If it yields success, test is considered 365 successful. <message> should state what it is testing. 366 367 Example: 368 369 test_expect_success \ 370 'git-write-tree should be able to write an empty tree.' \ 371 'tree=$(git-write-tree)' 372 373 If you supply three parameters the first will be taken to be a 374 prerequisite, see the test_set_prereq and test_have_prereq 375 documentation below: 376 377 test_expect_success TTY 'git --paginate rev-list uses a pager' \ 378 ' ... ' 379 380 You can also supply a comma-separated list of prerequisites, in the 381 rare case where your test depends on more than one: 382 383 test_expect_success PERL,PYTHON 'yo dawg' \ 384 ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' 385 386 - test_expect_failure [<prereq>] <message> <script> 387 388 This is NOT the opposite of test_expect_success, but is used 389 to mark a test that demonstrates a known breakage. Unlike 390 the usual test_expect_success tests, which say "ok" on 391 success and "FAIL" on failure, this will say "FIXED" on 392 success and "still broken" on failure. Failures from these 393 tests won't cause -i (immediate) to stop. 394 395 Like test_expect_success this function can optionally use a three 396 argument invocation with a prerequisite as the first argument. 397 398 - test_debug <script> 399 400 This takes a single argument, <script>, and evaluates it only 401 when the test script is started with --debug command line 402 argument. This is primarily meant for use during the 403 development of a new test script. 404 405 - test_done 406 407 Your test script must have test_done at the end. Its purpose 408 is to summarize successes and failures in the test script and 409 exit with an appropriate error code. 410 411 - test_tick 412 413 Make commit and tag names consistent by setting the author and 414 committer times to defined stated. Subsequent calls will 415 advance the times by a fixed amount. 416 417 - test_commit <message> [<filename> [<contents>]] 418 419 Creates a commit with the given message, committing the given 420 file with the given contents (default for both is to reuse the 421 message string), and adds a tag (again reusing the message 422 string as name). Calls test_tick to make the SHA-1s 423 reproducible. 424 425 - test_merge <message> <commit-or-tag> 426 427 Merges the given rev using the given message. Like test_commit, 428 creates a tag and calls test_tick before committing. 429 430 - test_set_prereq SOME_PREREQ 431 432 Set a test prerequisite to be used later with test_have_prereq. The 433 test-lib will set some prerequisites for you, see the 434 "Prerequisites" section below for a full list of these. 435 436 Others you can set yourself and use later with either 437 test_have_prereq directly, or the three argument invocation of 438 test_expect_success and test_expect_failure. 439 440 - test_have_prereq SOME PREREQ 441 442 Check if we have a prerequisite previously set with 443 test_set_prereq. The most common use of this directly is to skip 444 all the tests if we don't have some essential prerequisite: 445 446 if ! test_have_prereq PERL 447 then 448 skip_all='skipping perl interface tests, perl not available' 449 test_done 450 fi 451 452 - test_external [<prereq>] <message> <external> <script> 453 454 Execute a <script> with an <external> interpreter (like perl). This 455 was added for tests like t9700-perl-git.sh which do most of their 456 work in an external test script. 457 458 test_external \ 459 'GitwebCache::*FileCache*' \ 460 "$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_cache_interface.pl 461 462 If the test is outputting its own TAP you should set the 463 test_external_has_tap variable somewhere before calling the first 464 test_external* function. See t9700-perl-git.sh for an example. 465 466 # The external test will outputs its own plan 467 test_external_has_tap=1 468 469 - test_external_without_stderr [<prereq>] <message> <external> <script> 470 471 Like test_external but fail if there's any output on stderr, 472 instead of checking the exit code. 473 474 test_external_without_stderr \ 475 'Perl API' \ 476 "$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl 477 478 - test_expect_code <exit-code> <command> 479 480 Run a command and ensure that it exits with the given exit code. 481 For example: 482 483 test_expect_success 'Merge with d/f conflicts' ' 484 test_expect_code 1 git merge "merge msg" B master 485 ' 486 487 - test_must_fail <git-command> 488 489 Run a git command and ensure it fails in a controlled way. Use 490 this instead of "! <git-command>". When git-command dies due to a 491 segfault, test_must_fail diagnoses it as an error; "! <git-command>" 492 treats it as just another expected failure, which would let such a 493 bug go unnoticed. 494 495 - test_might_fail <git-command> 496 497 Similar to test_must_fail, but tolerate success, too. Use this 498 instead of "<git-command> || :" to catch failures due to segv. 499 500 - test_cmp <expected> <actual> 501 502 Check whether the content of the <actual> file matches the 503 <expected> file. This behaves like "cmp" but produces more 504 helpful output when the test is run with "-v" option. 505 506 - test_path_is_file <file> [<diagnosis>] 507 test_path_is_dir <dir> [<diagnosis>] 508 test_path_is_missing <path> [<diagnosis>] 509 510 Check whether a file/directory exists or doesn't. <diagnosis> will 511 be displayed if the test fails. 512 513 - test_when_finished <script> 514 515 Prepend <script> to a list of commands to run to clean up 516 at the end of the current test. If some clean-up command 517 fails, the test will not pass. 518 519 Example: 520 521 test_expect_success 'branch pointing to non-commit' ' 522 git rev-parse HEAD^{tree} >.git/refs/heads/invalid && 523 test_when_finished "git update-ref -d refs/heads/invalid" && 524 ... 525 ' 526 527Prerequisites 528------------- 529 530These are the prerequisites that the test library predefines with 531test_have_prereq. 532 533See the prereq argument to the test_* functions in the "Test harness 534library" section above and the "test_have_prereq" function for how to 535use these, and "test_set_prereq" for how to define your own. 536 537 - PERL & PYTHON 538 539 Git wasn't compiled with NO_PERL=YesPlease or 540 NO_PYTHON=YesPlease. Wrap any tests that need Perl or Python in 541 these. 542 543 - POSIXPERM 544 545 The filesystem supports POSIX style permission bits. 546 547 - BSLASHPSPEC 548 549 Backslashes in pathspec are not directory separators. This is not 550 set on Windows. See 6fd1106a for details. 551 552 - EXECKEEPSPID 553 554 The process retains the same pid across exec(2). See fb9a2bea for 555 details. 556 557 - SYMLINKS 558 559 The filesystem we're on supports symbolic links. E.g. a FAT 560 filesystem doesn't support these. See 704a3143 for details. 561 562 - SANITY 563 564 Test is not run by root user, and an attempt to write to an 565 unwritable file is expected to fail correctly. 566 567Tips for Writing Tests 568---------------------- 569 570As with any programming projects, existing programs are the best 571source of the information. However, do _not_ emulate 572t0000-basic.sh when writing your tests. The test is special in 573that it tries to validate the very core of GIT. For example, it 574knows that there will be 256 subdirectories under .git/objects/, 575and it knows that the object ID of an empty tree is a certain 57640-byte string. This is deliberately done so in t0000-basic.sh 577because the things the very basic core test tries to achieve is 578to serve as a basis for people who are changing the GIT internal 579drastically. For these people, after making certain changes, 580not seeing failures from the basic test _is_ a failure. And 581such drastic changes to the core GIT that even changes these 582otherwise supposedly stable object IDs should be accompanied by 583an update to t0000-basic.sh. 584 585However, other tests that simply rely on basic parts of the core 586GIT working properly should not have that level of intimate 587knowledge of the core GIT internals. If all the test scripts 588hardcoded the object IDs like t0000-basic.sh does, that defeats 589the purpose of t0000-basic.sh, which is to isolate that level of 590validation in one place. Your test also ends up needing 591updating when such a change to the internal happens, so do _not_ 592do it and leave the low level of validation to t0000-basic.sh. 593 594Test coverage 595------------- 596 597You can use the coverage tests to find code paths that are not being 598used or properly exercised yet. 599 600To do that, run the coverage target at the top-level (not in the t/ 601directory): 602 603 make coverage 604 605That'll compile Git with GCC's coverage arguments, and generate a test 606report with gcov after the tests finish. Running the coverage tests 607can take a while, since running the tests in parallel is incompatible 608with GCC's coverage mode. 609 610After the tests have run you can generate a list of untested 611functions: 612 613 make coverage-untested-functions 614 615You can also generate a detailed per-file HTML report using the 616Devel::Cover module. To install it do: 617 618 # On Debian or Ubuntu: 619 sudo aptitude install libdevel-cover-perl 620 621 # From the CPAN with cpanminus 622 curl -L http://cpanmin.us | perl - --sudo --self-upgrade 623 cpanm --sudo Devel::Cover 624 625Then, at the top-level: 626 627 make cover_db_html 628 629That'll generate a detailed cover report in the "cover_db_html" 630directory, which you can then copy to a webserver, or inspect locally 631in a browser. 632 633Smoke testing 634------------- 635 636The Git test suite has support for smoke testing. Smoke testing is 637when you submit the results of a test run to a central server for 638analysis and aggregation. 639 640Running a smoke tester is an easy and valuable way of contributing to 641Git development, particularly if you have access to an uncommon OS on 642obscure hardware. 643 644After building Git you can generate a smoke report like this in the 645"t" directory: 646 647 make clean smoke 648 649You can also pass arguments via the environment. This should make it 650faster: 651 652 GIT_TEST_OPTS='--root=/dev/shm' TEST_JOBS=10 make clean smoke 653 654The "smoke" target will run the Git test suite with Perl's 655"TAP::Harness" module, and package up the results in a .tar.gz archive 656with "TAP::Harness::Archive". The former is included with Perl v5.10.1 657or later, but you'll need to install the latter from the CPAN. See the 658"Test coverage" section above for how you might do that. 659 660Once the "smoke" target finishes you'll see a message like this: 661 662 TAP Archive created at <path to git>/t/test-results/git-smoke.tar.gz 663 664To upload the smoke report you need to have curl(1) installed, then 665do: 666 667 make smoke_report 668 669To upload the report anonymously. Hopefully that'll return something 670like "Reported #7 added.". 671 672If you're going to be uploading reports frequently please request a 673user account by E-Mailing gitsmoke@v.nix.is. Once you have a username 674and password you'll be able to do: 675 676 SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report 677 678You can also add an additional comment to attach to the report, and/or 679a comma separated list of tags: 680 681 SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> \ 682 SMOKE_COMMENT=<comment> SMOKE_TAGS=<tags> \ 683 make smoke_report 684 685Once the report is uploaded it'll be made available at 686http://smoke.git.nix.is, here's an overview of Recent Smoke Reports 687for Git: 688 689 http://smoke.git.nix.is/app/projects/smoke_reports/1 690 691The reports will also be mirrored to GitHub every few hours: 692 693 http://github.com/gitsmoke/smoke-reports 694 695The Smolder SQLite database is also mirrored and made available for 696download: 697 698 http://github.com/gitsmoke/smoke-database 699 700Note that the database includes hashed (with crypt()) user passwords 701and E-Mail addresses. Don't use a valuable password for the smoke 702service if you have an account, or an E-Mail address you don't want to 703be publicly known. The user accounts are just meant to be convenient 704labels, they're not meant to be secure.