1Core GIT Tests 2============== 3 4This directory holds many test scripts for core GIT tools. The 5first part of this short document describes how to run the tests 6and read their output. 7 8When fixing the tools or adding enhancements, you are strongly 9encouraged to add tests in this directory to cover what you are 10trying to fix or enhance. The later part of this short document 11describes how your test scripts should be organized. 12 13 14Running Tests 15------------- 16 17The easiest way to run tests is to say "make". This runs all 18the tests. 19 20 *** t0000-basic.sh *** 21 ok 1 - .git/objects should be empty after git init in an empty repo. 22 ok 2 - .git/objects should have 3 subdirectories. 23 ok 3 - success is reported like this 24 ... 25 ok 43 - very long name in the index handled sanely 26 # fixed 1 known breakage(s) 27 # still have 1 known breakage(s) 28 # passed all remaining 42 test(s) 29 1..43 30 *** t0001-init.sh *** 31 ok 1 - plain 32 ok 2 - plain with GIT_WORK_TREE 33 ok 3 - plain bare 34 35Since the tests all output TAP (see http://testanything.org) they can 36be run with any TAP harness. Here's an example of parallel testing 37powered by a recent version of prove(1): 38 39 $ prove --timer --jobs 15 ./t[0-9]*.sh 40 [19:17:33] ./t0005-signals.sh ................................... ok 36 ms 41 [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms 42 [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms 43 [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms 44 [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms 45 ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )=== 46 47prove and other harnesses come with a lot of useful options. The 48--state option in particular is very useful: 49 50 # Repeat until no more failures 51 $ prove -j 15 --state=failed,save ./t[0-9]*.sh 52 53You can give DEFAULT_TEST_TARGET=prove on the make command (or define it 54in config.mak) to cause "make test" to run tests under prove. 55GIT_PROVE_OPTS can be used to pass additional options, e.g. 56 57 $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--timer --jobs 16' test 58 59You can also run each test individually from command line, like this: 60 61 $ sh ./t3010-ls-files-killed-modified.sh 62 ok 1 - git update-index --add to add various paths. 63 ok 2 - git ls-files -k to show killed files. 64 ok 3 - validate git ls-files -k output. 65 ok 4 - git ls-files -m to show modified files. 66 ok 5 - validate git ls-files -m output. 67 # passed all 5 test(s) 68 1..5 69 70You can pass --verbose (or -v), --debug (or -d), and --immediate 71(or -i) command line argument to the test, or by setting GIT_TEST_OPTS 72appropriately before running "make". 73 74--verbose:: 75 This makes the test more verbose. Specifically, the 76 command being run and their output if any are also 77 output. 78 79--debug:: 80 This may help the person who is developing a new test. 81 It causes the command defined with test_debug to run. 82 The "trash" directory (used to store all temporary data 83 during testing) is not deleted even if there are no 84 failed tests so that you can inspect its contents after 85 the test finished. 86 87--immediate:: 88 This causes the test to immediately exit upon the first 89 failed test. Cleanup commands requested with 90 test_when_finished are not executed if the test failed, 91 in order to keep the state for inspection by the tester 92 to diagnose the bug. 93 94--long-tests:: 95 This causes additional long-running tests to be run (where 96 available), for more exhaustive testing. 97 98--valgrind:: 99 Execute all Git binaries with valgrind and exit with status 100 126 on errors (just like regular tests, this will only stop 101 the test script when running under -i). Valgrind errors 102 go to stderr, so you might want to pass the -v option, too. 103 104 Since it makes no sense to run the tests with --valgrind and 105 not see any output, this option implies --verbose. For 106 convenience, it also implies --tee. 107 108 Note that valgrind is run with the option --leak-check=no, 109 as the git process is short-lived and some errors are not 110 interesting. In order to run a single command under the same 111 conditions manually, you should set GIT_VALGRIND to point to 112 the 't/valgrind/' directory and use the commands under 113 't/valgrind/bin/'. 114 115--tee:: 116 In addition to printing the test output to the terminal, 117 write it to files named 't/test-results/$TEST_NAME.out'. 118 As the names depend on the tests' file names, it is safe to 119 run the tests with this option in parallel. 120 121--with-dashes:: 122 By default tests are run without dashed forms of 123 commands (like git-commit) in the PATH (it only uses 124 wrappers from ../bin-wrappers). Use this option to include 125 the build directory (..) in the PATH, which contains all 126 the dashed forms of commands. This option is currently 127 implied by other options like --valgrind and 128 GIT_TEST_INSTALLED. 129 130--root=<directory>:: 131 Create "trash" directories used to store all temporary data during 132 testing under <directory>, instead of the t/ directory. 133 Using this option with a RAM-based filesystem (such as tmpfs) 134 can massively speed up the test suite. 135 136You can also set the GIT_TEST_INSTALLED environment variable to 137the bindir of an existing git installation to test that installation. 138You still need to have built this git sandbox, from which various 139test-* support programs, templates, and perl libraries are used. 140If your installed git is incomplete, it will silently test parts of 141your built version instead. 142 143When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to 144override the location of the dashed-form subcommands (what 145GIT_EXEC_PATH would be used for during normal operation). 146GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. 147 148 149Skipping Tests 150-------------- 151 152In some environments, certain tests have no way of succeeding 153due to platform limitation, such as lack of 'unzip' program, or 154filesystem that do not allow arbitrary sequence of non-NUL bytes 155as pathnames. 156 157You should be able to say something like 158 159 $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh 160 161and even: 162 163 $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make 164 165to omit such tests. The value of the environment variable is a 166SP separated list of patterns that tells which tests to skip, 167and either can match the "t[0-9]{4}" part to skip the whole 168test, or t[0-9]{4} followed by ".$number" to say which 169particular test to skip. 170 171Note that some tests in the existing test suite rely on previous 172test item, so you cannot arbitrarily disable one and expect the 173remainder of test to check what the test originally was intended 174to check. 175 176 177Naming Tests 178------------ 179 180The test files are named as: 181 182 tNNNN-commandname-details.sh 183 184where N is a decimal digit. 185 186First digit tells the family: 187 188 0 - the absolute basics and global stuff 189 1 - the basic commands concerning database 190 2 - the basic commands concerning the working tree 191 3 - the other basic commands (e.g. ls-files) 192 4 - the diff commands 193 5 - the pull and exporting commands 194 6 - the revision tree commands (even e.g. merge-base) 195 7 - the porcelainish commands concerning the working tree 196 8 - the porcelainish commands concerning forensics 197 9 - the git tools 198 199Second digit tells the particular command we are testing. 200 201Third digit (optionally) tells the particular switch or group of switches 202we are testing. 203 204If you create files under t/ directory (i.e. here) that is not 205the top-level test script, never name the file to match the above 206pattern. The Makefile here considers all such files as the 207top-level test script and tries to run all of them. Care is 208especially needed if you are creating a common test library 209file, similar to test-lib.sh, because such a library file may 210not be suitable for standalone execution. 211 212 213Writing Tests 214------------- 215 216The test script is written as a shell script. It should start 217with the standard "#!/bin/sh" with copyright notices, and an 218assignment to variable 'test_description', like this: 219 220 #!/bin/sh 221 # 222 # Copyright (c) 2005 Junio C Hamano 223 # 224 225 test_description='xxx test (option --frotz) 226 227 This test registers the following structure in the cache 228 and tries to run git-ls-files with option --frotz.' 229 230 231Source 'test-lib.sh' 232-------------------- 233 234After assigning test_description, the test script should source 235test-lib.sh like this: 236 237 . ./test-lib.sh 238 239This test harness library does the following things: 240 241 - If the script is invoked with command line argument --help 242 (or -h), it shows the test_description and exits. 243 244 - Creates an empty test directory with an empty .git/objects database 245 and chdir(2) into it. This directory is 't/trash 246 directory.$test_name_without_dotsh', with t/ subject to change by 247 the --root option documented above. 248 249 - Defines standard test helper functions for your scripts to 250 use. These functions are designed to make all scripts behave 251 consistently when command line arguments --verbose (or -v), 252 --debug (or -d), and --immediate (or -i) is given. 253 254Do's, don'ts & things to keep in mind 255------------------------------------- 256 257Here are a few examples of things you probably should and shouldn't do 258when writing tests. 259 260Do: 261 262 - Put all code inside test_expect_success and other assertions. 263 264 Even code that isn't a test per se, but merely some setup code 265 should be inside a test assertion. 266 267 - Chain your test assertions 268 269 Write test code like this: 270 271 git merge foo && 272 git push bar && 273 test ... 274 275 Instead of: 276 277 git merge hla 278 git push gh 279 test ... 280 281 That way all of the commands in your tests will succeed or fail. If 282 you must ignore the return value of something, consider using a 283 helper function (e.g. use sane_unset instead of unset, in order 284 to avoid unportable return value for unsetting a variable that was 285 already unset), or prepending the command with test_might_fail or 286 test_must_fail. 287 288 - Check the test coverage for your tests. See the "Test coverage" 289 below. 290 291 Don't blindly follow test coverage metrics; if a new function you added 292 doesn't have any coverage, then you're probably doing something wrong, 293 but having 100% coverage doesn't necessarily mean that you tested 294 everything. 295 296 Tests that are likely to smoke out future regressions are better 297 than tests that just inflate the coverage metrics. 298 299 - When a test checks for an absolute path that a git command generated, 300 construct the expected value using $(pwd) rather than $PWD, 301 $TEST_DIRECTORY, or $TRASH_DIRECTORY. It makes a difference on 302 Windows, where the shell (MSYS bash) mangles absolute path names. 303 For details, see the commit message of 4114156ae9. 304 305Don't: 306 307 - exit() within a <script> part. 308 309 The harness will catch this as a programming error of the test. 310 Use test_done instead if you need to stop the tests early (see 311 "Skipping tests" below). 312 313 - use '! git cmd' when you want to make sure the git command exits 314 with failure in a controlled way by calling "die()". Instead, 315 use 'test_must_fail git cmd'. This will signal a failure if git 316 dies in an unexpected way (e.g. segfault). 317 318 - use perl without spelling it as "$PERL_PATH". This is to help our 319 friends on Windows where the platform Perl often adds CR before 320 the end of line, and they bundle Git with a version of Perl that 321 does not do so, whose path is specified with $PERL_PATH. 322 323 - use sh without spelling it as "$SHELL_PATH", when the script can 324 be misinterpreted by broken platform shell (e.g. Solaris). 325 326 - chdir around in tests. It is not sufficient to chdir to 327 somewhere and then chdir back to the original location later in 328 the test, as any intermediate step can fail and abort the test, 329 causing the next test to start in an unexpected directory. Do so 330 inside a subshell if necessary. 331 332 - Break the TAP output 333 334 The raw output from your test may be interpreted by a TAP harness. TAP 335 harnesses will ignore everything they don't know about, but don't step 336 on their toes in these areas: 337 338 - Don't print lines like "$x..$y" where $x and $y are integers. 339 340 - Don't print lines that begin with "ok" or "not ok". 341 342 TAP harnesses expect a line that begins with either "ok" and "not 343 ok" to signal a test passed or failed (and our harness already 344 produces such lines), so your script shouldn't emit such lines to 345 their output. 346 347 You can glean some further possible issues from the TAP grammar 348 (see http://search.cpan.org/perldoc?TAP::Parser::Grammar#TAP_Grammar) 349 but the best indication is to just run the tests with prove(1), 350 it'll complain if anything is amiss. 351 352Keep in mind: 353 354 - Inside <script> part, the standard output and standard error 355 streams are discarded, and the test harness only reports "ok" or 356 "not ok" to the end user running the tests. Under --verbose, they 357 are shown to help debugging the tests. 358 359 360Skipping tests 361-------------- 362 363If you need to skip tests you should do so by using the three-arg form 364of the test_* functions (see the "Test harness library" section 365below), e.g.: 366 367 test_expect_success PERL 'I need Perl' ' 368 "$PERL_PATH" -e "hlagh() if unf_unf()" 369 ' 370 371The advantage of skipping tests like this is that platforms that don't 372have the PERL and other optional dependencies get an indication of how 373many tests they're missing. 374 375If the test code is too hairy for that (i.e. does a lot of setup work 376outside test assertions) you can also skip all remaining tests by 377setting skip_all and immediately call test_done: 378 379 if ! test_have_prereq PERL 380 then 381 skip_all='skipping perl interface tests, perl not available' 382 test_done 383 fi 384 385The string you give to skip_all will be used as an explanation for why 386the test was skipped. 387 388End with test_done 389------------------ 390 391Your script will be a sequence of tests, using helper functions 392from the test harness library. At the end of the script, call 393'test_done'. 394 395 396Test harness library 397-------------------- 398 399There are a handful helper functions defined in the test harness 400library for your script to use. 401 402 - test_expect_success [<prereq>] <message> <script> 403 404 Usually takes two strings as parameters, and evaluates the 405 <script>. If it yields success, test is considered 406 successful. <message> should state what it is testing. 407 408 Example: 409 410 test_expect_success \ 411 'git-write-tree should be able to write an empty tree.' \ 412 'tree=$(git-write-tree)' 413 414 If you supply three parameters the first will be taken to be a 415 prerequisite; see the test_set_prereq and test_have_prereq 416 documentation below: 417 418 test_expect_success TTY 'git --paginate rev-list uses a pager' \ 419 ' ... ' 420 421 You can also supply a comma-separated list of prerequisites, in the 422 rare case where your test depends on more than one: 423 424 test_expect_success PERL,PYTHON 'yo dawg' \ 425 ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' 426 427 - test_expect_failure [<prereq>] <message> <script> 428 429 This is NOT the opposite of test_expect_success, but is used 430 to mark a test that demonstrates a known breakage. Unlike 431 the usual test_expect_success tests, which say "ok" on 432 success and "FAIL" on failure, this will say "FIXED" on 433 success and "still broken" on failure. Failures from these 434 tests won't cause -i (immediate) to stop. 435 436 Like test_expect_success this function can optionally use a three 437 argument invocation with a prerequisite as the first argument. 438 439 - test_debug <script> 440 441 This takes a single argument, <script>, and evaluates it only 442 when the test script is started with --debug command line 443 argument. This is primarily meant for use during the 444 development of a new test script. 445 446 - test_done 447 448 Your test script must have test_done at the end. Its purpose 449 is to summarize successes and failures in the test script and 450 exit with an appropriate error code. 451 452 - test_tick 453 454 Make commit and tag names consistent by setting the author and 455 committer times to defined state. Subsequent calls will 456 advance the times by a fixed amount. 457 458 - test_commit <message> [<filename> [<contents>]] 459 460 Creates a commit with the given message, committing the given 461 file with the given contents (default for both is to reuse the 462 message string), and adds a tag (again reusing the message 463 string as name). Calls test_tick to make the SHA-1s 464 reproducible. 465 466 - test_merge <message> <commit-or-tag> 467 468 Merges the given rev using the given message. Like test_commit, 469 creates a tag and calls test_tick before committing. 470 471 - test_set_prereq <prereq> 472 473 Set a test prerequisite to be used later with test_have_prereq. The 474 test-lib will set some prerequisites for you, see the 475 "Prerequisites" section below for a full list of these. 476 477 Others you can set yourself and use later with either 478 test_have_prereq directly, or the three argument invocation of 479 test_expect_success and test_expect_failure. 480 481 - test_have_prereq <prereq> 482 483 Check if we have a prerequisite previously set with 484 test_set_prereq. The most common use of this directly is to skip 485 all the tests if we don't have some essential prerequisite: 486 487 if ! test_have_prereq PERL 488 then 489 skip_all='skipping perl interface tests, perl not available' 490 test_done 491 fi 492 493 - test_external [<prereq>] <message> <external> <script> 494 495 Execute a <script> with an <external> interpreter (like perl). This 496 was added for tests like t9700-perl-git.sh which do most of their 497 work in an external test script. 498 499 test_external \ 500 'GitwebCache::*FileCache*' \ 501 "$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_cache_interface.pl 502 503 If the test is outputting its own TAP you should set the 504 test_external_has_tap variable somewhere before calling the first 505 test_external* function. See t9700-perl-git.sh for an example. 506 507 # The external test will outputs its own plan 508 test_external_has_tap=1 509 510 - test_external_without_stderr [<prereq>] <message> <external> <script> 511 512 Like test_external but fail if there's any output on stderr, 513 instead of checking the exit code. 514 515 test_external_without_stderr \ 516 'Perl API' \ 517 "$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl 518 519 - test_expect_code <exit-code> <command> 520 521 Run a command and ensure that it exits with the given exit code. 522 For example: 523 524 test_expect_success 'Merge with d/f conflicts' ' 525 test_expect_code 1 git merge "merge msg" B master 526 ' 527 528 - test_must_fail <git-command> 529 530 Run a git command and ensure it fails in a controlled way. Use 531 this instead of "! <git-command>". When git-command dies due to a 532 segfault, test_must_fail diagnoses it as an error; "! <git-command>" 533 treats it as just another expected failure, which would let such a 534 bug go unnoticed. 535 536 - test_might_fail <git-command> 537 538 Similar to test_must_fail, but tolerate success, too. Use this 539 instead of "<git-command> || :" to catch failures due to segv. 540 541 - test_cmp <expected> <actual> 542 543 Check whether the content of the <actual> file matches the 544 <expected> file. This behaves like "cmp" but produces more 545 helpful output when the test is run with "-v" option. 546 547 - test_line_count (= | -lt | -ge | ...) <length> <file> 548 549 Check whether a file has the length it is expected to. 550 551 - test_path_is_file <path> [<diagnosis>] 552 test_path_is_dir <path> [<diagnosis>] 553 test_path_is_missing <path> [<diagnosis>] 554 555 Check if the named path is a file, if the named path is a 556 directory, or if the named path does not exist, respectively, 557 and fail otherwise, showing the <diagnosis> text. 558 559 - test_when_finished <script> 560 561 Prepend <script> to a list of commands to run to clean up 562 at the end of the current test. If some clean-up command 563 fails, the test will not pass. 564 565 Example: 566 567 test_expect_success 'branch pointing to non-commit' ' 568 git rev-parse HEAD^{tree} >.git/refs/heads/invalid && 569 test_when_finished "git update-ref -d refs/heads/invalid" && 570 ... 571 ' 572 573 - test_pause 574 575 This command is useful for writing and debugging tests and must be 576 removed before submitting. It halts the execution of the test and 577 spawns a shell in the trash directory. Exit the shell to continue 578 the test. Example: 579 580 test_expect_success 'test' ' 581 git do-something >actual && 582 test_pause && 583 test_cmp expected actual 584 ' 585 586Prerequisites 587------------- 588 589These are the prerequisites that the test library predefines with 590test_have_prereq. 591 592See the prereq argument to the test_* functions in the "Test harness 593library" section above and the "test_have_prereq" function for how to 594use these, and "test_set_prereq" for how to define your own. 595 596 - PERL & PYTHON 597 598 Git wasn't compiled with NO_PERL=YesPlease or 599 NO_PYTHON=YesPlease. Wrap any tests that need Perl or Python in 600 these. 601 602 - POSIXPERM 603 604 The filesystem supports POSIX style permission bits. 605 606 - BSLASHPSPEC 607 608 Backslashes in pathspec are not directory separators. This is not 609 set on Windows. See 6fd1106a for details. 610 611 - EXECKEEPSPID 612 613 The process retains the same pid across exec(2). See fb9a2bea for 614 details. 615 616 - PIPE 617 618 The filesystem we're on supports creation of FIFOs (named pipes) 619 via mkfifo(1). 620 621 - SYMLINKS 622 623 The filesystem we're on supports symbolic links. E.g. a FAT 624 filesystem doesn't support these. See 704a3143 for details. 625 626 - SANITY 627 628 Test is not run by root user, and an attempt to write to an 629 unwritable file is expected to fail correctly. 630 631 - LIBPCRE 632 633 Git was compiled with USE_LIBPCRE=YesPlease. Wrap any tests 634 that use git-grep --perl-regexp or git-grep -P in these. 635 636 - CASE_INSENSITIVE_FS 637 638 Test is run on a case insensitive file system. 639 640 - UTF8_NFD_TO_NFC 641 642 Test is run on a filesystem which converts decomposed utf-8 (nfd) 643 to precomposed utf-8 (nfc). 644 645Tips for Writing Tests 646---------------------- 647 648As with any programming projects, existing programs are the best 649source of the information. However, do _not_ emulate 650t0000-basic.sh when writing your tests. The test is special in 651that it tries to validate the very core of GIT. For example, it 652knows that there will be 256 subdirectories under .git/objects/, 653and it knows that the object ID of an empty tree is a certain 65440-byte string. This is deliberately done so in t0000-basic.sh 655because the things the very basic core test tries to achieve is 656to serve as a basis for people who are changing the GIT internal 657drastically. For these people, after making certain changes, 658not seeing failures from the basic test _is_ a failure. And 659such drastic changes to the core GIT that even changes these 660otherwise supposedly stable object IDs should be accompanied by 661an update to t0000-basic.sh. 662 663However, other tests that simply rely on basic parts of the core 664GIT working properly should not have that level of intimate 665knowledge of the core GIT internals. If all the test scripts 666hardcoded the object IDs like t0000-basic.sh does, that defeats 667the purpose of t0000-basic.sh, which is to isolate that level of 668validation in one place. Your test also ends up needing 669updating when such a change to the internal happens, so do _not_ 670do it and leave the low level of validation to t0000-basic.sh. 671 672Test coverage 673------------- 674 675You can use the coverage tests to find code paths that are not being 676used or properly exercised yet. 677 678To do that, run the coverage target at the top-level (not in the t/ 679directory): 680 681 make coverage 682 683That'll compile Git with GCC's coverage arguments, and generate a test 684report with gcov after the tests finish. Running the coverage tests 685can take a while, since running the tests in parallel is incompatible 686with GCC's coverage mode. 687 688After the tests have run you can generate a list of untested 689functions: 690 691 make coverage-untested-functions 692 693You can also generate a detailed per-file HTML report using the 694Devel::Cover module. To install it do: 695 696 # On Debian or Ubuntu: 697 sudo aptitude install libdevel-cover-perl 698 699 # From the CPAN with cpanminus 700 curl -L http://cpanmin.us | perl - --sudo --self-upgrade 701 cpanm --sudo Devel::Cover 702 703Then, at the top-level: 704 705 make cover_db_html 706 707That'll generate a detailed cover report in the "cover_db_html" 708directory, which you can then copy to a webserver, or inspect locally 709in a browser.