1Core GIT Tests 2============== 3 4This directory holds many test scripts for core GIT tools. The 5first part of this short document describes how to run the tests 6and read their output. 7 8When fixing the tools or adding enhancements, you are strongly 9encouraged to add tests in this directory to cover what you are 10trying to fix or enhance. The later part of this short document 11describes how your test scripts should be organized. 12 13 14Running Tests 15------------- 16 17The easiest way to run tests is to say "make". This runs all 18the tests. 19 20 *** t0000-basic.sh *** 21 ok 1 - .git/objects should be empty after git init in an empty repo. 22 ok 2 - .git/objects should have 3 subdirectories. 23 ok 3 - success is reported like this 24 ... 25 ok 43 - very long name in the index handled sanely 26 # fixed 1 known breakage(s) 27 # still have 1 known breakage(s) 28 # passed all remaining 42 test(s) 29 1..43 30 *** t0001-init.sh *** 31 ok 1 - plain 32 ok 2 - plain with GIT_WORK_TREE 33 ok 3 - plain bare 34 35Since the tests all output TAP (see http://testanything.org) they can 36be run with any TAP harness. Here's an example of parallel testing 37powered by a recent version of prove(1): 38 39 $ prove --timer --jobs 15 ./t[0-9]*.sh 40 [19:17:33] ./t0005-signals.sh ................................... ok 36 ms 41 [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms 42 [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms 43 [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms 44 [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms 45 ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )=== 46 47prove and other harnesses come with a lot of useful options. The 48--state option in particular is very useful: 49 50 # Repeat until no more failures 51 $ prove -j 15 --state=failed,save ./t[0-9]*.sh 52 53You can also run each test individually from command line, like this: 54 55 $ sh ./t3010-ls-files-killed-modified.sh 56 ok 1 - git update-index --add to add various paths. 57 ok 2 - git ls-files -k to show killed files. 58 ok 3 - validate git ls-files -k output. 59 ok 4 - git ls-files -m to show modified files. 60 ok 5 - validate git ls-files -m output. 61 # passed all 5 test(s) 62 1..5 63 64You can pass --verbose (or -v), --debug (or -d), and --immediate 65(or -i) command line argument to the test, or by setting GIT_TEST_OPTS 66appropriately before running "make". 67 68--verbose:: 69 This makes the test more verbose. Specifically, the 70 command being run and their output if any are also 71 output. 72 73--debug:: 74 This may help the person who is developing a new test. 75 It causes the command defined with test_debug to run. 76 77--immediate:: 78 This causes the test to immediately exit upon the first 79 failed test. 80 81--long-tests:: 82 This causes additional long-running tests to be run (where 83 available), for more exhaustive testing. 84 85--valgrind:: 86 Execute all Git binaries with valgrind and exit with status 87 126 on errors (just like regular tests, this will only stop 88 the test script when running under -i). Valgrind errors 89 go to stderr, so you might want to pass the -v option, too. 90 91 Since it makes no sense to run the tests with --valgrind and 92 not see any output, this option implies --verbose. For 93 convenience, it also implies --tee. 94 95--tee:: 96 In addition to printing the test output to the terminal, 97 write it to files named 't/test-results/$TEST_NAME.out'. 98 As the names depend on the tests' file names, it is safe to 99 run the tests with this option in parallel. 100 101--with-dashes:: 102 By default tests are run without dashed forms of 103 commands (like git-commit) in the PATH (it only uses 104 wrappers from ../bin-wrappers). Use this option to include 105 the build directory (..) in the PATH, which contains all 106 the dashed forms of commands. This option is currently 107 implied by other options like --valgrind and 108 GIT_TEST_INSTALLED. 109 110--root=<directory>:: 111 Create "trash" directories used to store all temporary data during 112 testing under <directory>, instead of the t/ directory. 113 Using this option with a RAM-based filesystem (such as tmpfs) 114 can massively speed up the test suite. 115 116You can also set the GIT_TEST_INSTALLED environment variable to 117the bindir of an existing git installation to test that installation. 118You still need to have built this git sandbox, from which various 119test-* support programs, templates, and perl libraries are used. 120If your installed git is incomplete, it will silently test parts of 121your built version instead. 122 123When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to 124override the location of the dashed-form subcommands (what 125GIT_EXEC_PATH would be used for during normal operation). 126GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. 127 128 129Skipping Tests 130-------------- 131 132In some environments, certain tests have no way of succeeding 133due to platform limitation, such as lack of 'unzip' program, or 134filesystem that do not allow arbitrary sequence of non-NUL bytes 135as pathnames. 136 137You should be able to say something like 138 139 $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh 140 141and even: 142 143 $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make 144 145to omit such tests. The value of the environment variable is a 146SP separated list of patterns that tells which tests to skip, 147and either can match the "t[0-9]{4}" part to skip the whole 148test, or t[0-9]{4} followed by ".$number" to say which 149particular test to skip. 150 151Note that some tests in the existing test suite rely on previous 152test item, so you cannot arbitrarily disable one and expect the 153remainder of test to check what the test originally was intended 154to check. 155 156 157Naming Tests 158------------ 159 160The test files are named as: 161 162 tNNNN-commandname-details.sh 163 164where N is a decimal digit. 165 166First digit tells the family: 167 168 0 - the absolute basics and global stuff 169 1 - the basic commands concerning database 170 2 - the basic commands concerning the working tree 171 3 - the other basic commands (e.g. ls-files) 172 4 - the diff commands 173 5 - the pull and exporting commands 174 6 - the revision tree commands (even e.g. merge-base) 175 7 - the porcelainish commands concerning the working tree 176 8 - the porcelainish commands concerning forensics 177 9 - the git tools 178 179Second digit tells the particular command we are testing. 180 181Third digit (optionally) tells the particular switch or group of switches 182we are testing. 183 184If you create files under t/ directory (i.e. here) that is not 185the top-level test script, never name the file to match the above 186pattern. The Makefile here considers all such files as the 187top-level test script and tries to run all of them. A care is 188especially needed if you are creating a common test library 189file, similar to test-lib.sh, because such a library file may 190not be suitable for standalone execution. 191 192 193Writing Tests 194------------- 195 196The test script is written as a shell script. It should start 197with the standard "#!/bin/sh" with copyright notices, and an 198assignment to variable 'test_description', like this: 199 200 #!/bin/sh 201 # 202 # Copyright (c) 2005 Junio C Hamano 203 # 204 205 test_description='xxx test (option --frotz) 206 207 This test registers the following structure in the cache 208 and tries to run git-ls-files with option --frotz.' 209 210 211Source 'test-lib.sh' 212-------------------- 213 214After assigning test_description, the test script should source 215test-lib.sh like this: 216 217 . ./test-lib.sh 218 219This test harness library does the following things: 220 221 - If the script is invoked with command line argument --help 222 (or -h), it shows the test_description and exits. 223 224 - Creates an empty test directory with an empty .git/objects database 225 and chdir(2) into it. This directory is 't/trash 226 directory.$test_name_without_dotsh', with t/ subject to change by 227 the --root option documented above. 228 229 - Defines standard test helper functions for your scripts to 230 use. These functions are designed to make all scripts behave 231 consistently when command line arguments --verbose (or -v), 232 --debug (or -d), and --immediate (or -i) is given. 233 234Do's, don'ts & things to keep in mind 235------------------------------------- 236 237Here are a few examples of things you probably should and shouldn't do 238when writing tests. 239 240Do: 241 242 - Put all code inside test_expect_success and other assertions. 243 244 Even code that isn't a test per se, but merely some setup code 245 should be inside a test assertion. 246 247 - Chain your test assertions 248 249 Write test code like this: 250 251 git merge foo && 252 git push bar && 253 test ... 254 255 Instead of: 256 257 git merge hla 258 git push gh 259 test ... 260 261 That way all of the commands in your tests will succeed or fail. If 262 you must ignore the return value of something, consider using a 263 helper function (e.g. use sane_unset instead of unset, in order 264 to avoid unportable return value for unsetting a variable that was 265 already unset), or prepending the command with test_might_fail or 266 test_must_fail. 267 268 - Check the test coverage for your tests. See the "Test coverage" 269 below. 270 271 Don't blindly follow test coverage metrics, they're a good way to 272 spot if you've missed something. If a new function you added 273 doesn't have any coverage you're probably doing something wrong, 274 but having 100% coverage doesn't necessarily mean that you tested 275 everything. 276 277 Tests that are likely to smoke out future regressions are better 278 than tests that just inflate the coverage metrics. 279 280Don't: 281 282 - exit() within a <script> part. 283 284 The harness will catch this as a programming error of the test. 285 Use test_done instead if you need to stop the tests early (see 286 "Skipping tests" below). 287 288 - Break the TAP output 289 290 The raw output from your test may be interpreted by a TAP harness. TAP 291 harnesses will ignore everything they don't know about, but don't step 292 on their toes in these areas: 293 294 - Don't print lines like "$x..$y" where $x and $y are integers. 295 296 - Don't print lines that begin with "ok" or "not ok". 297 298 TAP harnesses expect a line that begins with either "ok" and "not 299 ok" to signal a test passed or failed (and our harness already 300 produces such lines), so your script shouldn't emit such lines to 301 their output. 302 303 You can glean some further possible issues from the TAP grammar 304 (see http://search.cpan.org/perldoc?TAP::Parser::Grammar#TAP_Grammar) 305 but the best indication is to just run the tests with prove(1), 306 it'll complain if anything is amiss. 307 308Keep in mind: 309 310 - Inside <script> part, the standard output and standard error 311 streams are discarded, and the test harness only reports "ok" or 312 "not ok" to the end user running the tests. Under --verbose, they 313 are shown to help debugging the tests. 314 315 316Skipping tests 317-------------- 318 319If you need to skip tests you should do so be using the three-arg form 320of the test_* functions (see the "Test harness library" section 321below), e.g.: 322 323 test_expect_success PERL 'I need Perl' " 324 '$PERL_PATH' -e 'hlagh() if unf_unf()' 325 " 326 327The advantage of skipping tests like this is that platforms that don't 328have the PERL and other optional dependencies get an indication of how 329many tests they're missing. 330 331If the test code is too hairy for that (i.e. does a lot of setup work 332outside test assertions) you can also skip all remaining tests by 333setting skip_all and immediately call test_done: 334 335 if ! test_have_prereq PERL 336 then 337 skip_all='skipping perl interface tests, perl not available' 338 test_done 339 fi 340 341The string you give to skip_all will be used as an explanation for why 342the test was skipped. 343 344End with test_done 345------------------ 346 347Your script will be a sequence of tests, using helper functions 348from the test harness library. At the end of the script, call 349'test_done'. 350 351 352Test harness library 353-------------------- 354 355There are a handful helper functions defined in the test harness 356library for your script to use. 357 358 - test_expect_success [<prereq>] <message> <script> 359 360 Usually takes two strings as parameter, and evaluates the 361 <script>. If it yields success, test is considered 362 successful. <message> should state what it is testing. 363 364 Example: 365 366 test_expect_success \ 367 'git-write-tree should be able to write an empty tree.' \ 368 'tree=$(git-write-tree)' 369 370 If you supply three parameters the first will be taken to be a 371 prerequisite, see the test_set_prereq and test_have_prereq 372 documentation below: 373 374 test_expect_success TTY 'git --paginate rev-list uses a pager' \ 375 ' ... ' 376 377 You can also supply a comma-separated list of prerequisites, in the 378 rare case where your test depends on more than one: 379 380 test_expect_success PERL,PYTHON 'yo dawg' \ 381 ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' 382 383 - test_expect_failure [<prereq>] <message> <script> 384 385 This is NOT the opposite of test_expect_success, but is used 386 to mark a test that demonstrates a known breakage. Unlike 387 the usual test_expect_success tests, which say "ok" on 388 success and "FAIL" on failure, this will say "FIXED" on 389 success and "still broken" on failure. Failures from these 390 tests won't cause -i (immediate) to stop. 391 392 Like test_expect_success this function can optionally use a three 393 argument invocation with a prerequisite as the first argument. 394 395 - test_debug <script> 396 397 This takes a single argument, <script>, and evaluates it only 398 when the test script is started with --debug command line 399 argument. This is primarily meant for use during the 400 development of a new test script. 401 402 - test_done 403 404 Your test script must have test_done at the end. Its purpose 405 is to summarize successes and failures in the test script and 406 exit with an appropriate error code. 407 408 - test_tick 409 410 Make commit and tag names consistent by setting the author and 411 committer times to defined stated. Subsequent calls will 412 advance the times by a fixed amount. 413 414 - test_commit <message> [<filename> [<contents>]] 415 416 Creates a commit with the given message, committing the given 417 file with the given contents (default for both is to reuse the 418 message string), and adds a tag (again reusing the message 419 string as name). Calls test_tick to make the SHA-1s 420 reproducible. 421 422 - test_merge <message> <commit-or-tag> 423 424 Merges the given rev using the given message. Like test_commit, 425 creates a tag and calls test_tick before committing. 426 427 - test_set_prereq SOME_PREREQ 428 429 Set a test prerequisite to be used later with test_have_prereq. The 430 test-lib will set some prerequisites for you, see the 431 "Prerequisites" section below for a full list of these. 432 433 Others you can set yourself and use later with either 434 test_have_prereq directly, or the three argument invocation of 435 test_expect_success and test_expect_failure. 436 437 - test_have_prereq SOME PREREQ 438 439 Check if we have a prerequisite previously set with 440 test_set_prereq. The most common use of this directly is to skip 441 all the tests if we don't have some essential prerequisite: 442 443 if ! test_have_prereq PERL 444 then 445 skip_all='skipping perl interface tests, perl not available' 446 test_done 447 fi 448 449 - test_external [<prereq>] <message> <external> <script> 450 451 Execute a <script> with an <external> interpreter (like perl). This 452 was added for tests like t9700-perl-git.sh which do most of their 453 work in an external test script. 454 455 test_external \ 456 'GitwebCache::*FileCache*' \ 457 "$PERL_PATH" "$TEST_DIRECTORY"/t9503/test_cache_interface.pl 458 459 If the test is outputting its own TAP you should set the 460 test_external_has_tap variable somewhere before calling the first 461 test_external* function. See t9700-perl-git.sh for an example. 462 463 # The external test will outputs its own plan 464 test_external_has_tap=1 465 466 - test_external_without_stderr [<prereq>] <message> <external> <script> 467 468 Like test_external but fail if there's any output on stderr, 469 instead of checking the exit code. 470 471 test_external_without_stderr \ 472 'Perl API' \ 473 "$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl 474 475 - test_expect_code <exit-code> <command> 476 477 Run a command and ensure that it exits with the given exit code. 478 For example: 479 480 test_expect_success 'Merge with d/f conflicts' ' 481 test_expect_code 1 git merge "merge msg" B master 482 ' 483 484 - test_must_fail <git-command> 485 486 Run a git command and ensure it fails in a controlled way. Use 487 this instead of "! <git-command>". When git-command dies due to a 488 segfault, test_must_fail diagnoses it as an error; "! <git-command>" 489 treats it as just another expected failure, which would let such a 490 bug go unnoticed. 491 492 - test_might_fail <git-command> 493 494 Similar to test_must_fail, but tolerate success, too. Use this 495 instead of "<git-command> || :" to catch failures due to segv. 496 497 - test_cmp <expected> <actual> 498 499 Check whether the content of the <actual> file matches the 500 <expected> file. This behaves like "cmp" but produces more 501 helpful output when the test is run with "-v" option. 502 503 - test_path_is_file <file> [<diagnosis>] 504 test_path_is_dir <dir> [<diagnosis>] 505 test_path_is_missing <path> [<diagnosis>] 506 507 Check whether a file/directory exists or doesn't. <diagnosis> will 508 be displayed if the test fails. 509 510 - test_when_finished <script> 511 512 Prepend <script> to a list of commands to run to clean up 513 at the end of the current test. If some clean-up command 514 fails, the test will not pass. 515 516 Example: 517 518 test_expect_success 'branch pointing to non-commit' ' 519 git rev-parse HEAD^{tree} >.git/refs/heads/invalid && 520 test_when_finished "git update-ref -d refs/heads/invalid" && 521 ... 522 ' 523 524Prerequisites 525------------- 526 527These are the prerequisites that the test library predefines with 528test_have_prereq. 529 530See the prereq argument to the test_* functions in the "Test harness 531library" section above and the "test_have_prereq" function for how to 532use these, and "test_set_prereq" for how to define your own. 533 534 - PERL & PYTHON 535 536 Git wasn't compiled with NO_PERL=YesPlease or 537 NO_PYTHON=YesPlease. Wrap any tests that need Perl or Python in 538 these. 539 540 - POSIXPERM 541 542 The filesystem supports POSIX style permission bits. 543 544 - BSLASHPSPEC 545 546 Backslashes in pathspec are not directory separators. This is not 547 set on Windows. See 6fd1106a for details. 548 549 - EXECKEEPSPID 550 551 The process retains the same pid across exec(2). See fb9a2bea for 552 details. 553 554 - SYMLINKS 555 556 The filesystem we're on supports symbolic links. E.g. a FAT 557 filesystem doesn't support these. See 704a3143 for details. 558 559 - SANITY 560 561 Test is not run by root user, and an attempt to write to an 562 unwritable file is expected to fail correctly. 563 564Tips for Writing Tests 565---------------------- 566 567As with any programming projects, existing programs are the best 568source of the information. However, do _not_ emulate 569t0000-basic.sh when writing your tests. The test is special in 570that it tries to validate the very core of GIT. For example, it 571knows that there will be 256 subdirectories under .git/objects/, 572and it knows that the object ID of an empty tree is a certain 57340-byte string. This is deliberately done so in t0000-basic.sh 574because the things the very basic core test tries to achieve is 575to serve as a basis for people who are changing the GIT internal 576drastically. For these people, after making certain changes, 577not seeing failures from the basic test _is_ a failure. And 578such drastic changes to the core GIT that even changes these 579otherwise supposedly stable object IDs should be accompanied by 580an update to t0000-basic.sh. 581 582However, other tests that simply rely on basic parts of the core 583GIT working properly should not have that level of intimate 584knowledge of the core GIT internals. If all the test scripts 585hardcoded the object IDs like t0000-basic.sh does, that defeats 586the purpose of t0000-basic.sh, which is to isolate that level of 587validation in one place. Your test also ends up needing 588updating when such a change to the internal happens, so do _not_ 589do it and leave the low level of validation to t0000-basic.sh. 590 591Test coverage 592------------- 593 594You can use the coverage tests to find code paths that are not being 595used or properly exercised yet. 596 597To do that, run the coverage target at the top-level (not in the t/ 598directory): 599 600 make coverage 601 602That'll compile Git with GCC's coverage arguments, and generate a test 603report with gcov after the tests finish. Running the coverage tests 604can take a while, since running the tests in parallel is incompatible 605with GCC's coverage mode. 606 607After the tests have run you can generate a list of untested 608functions: 609 610 make coverage-untested-functions 611 612You can also generate a detailed per-file HTML report using the 613Devel::Cover module. To install it do: 614 615 # On Debian or Ubuntu: 616 sudo aptitude install libdevel-cover-perl 617 618 # From the CPAN with cpanminus 619 curl -L http://cpanmin.us | perl - --sudo --self-upgrade 620 cpanm --sudo Devel::Cover 621 622Then, at the top-level: 623 624 make cover_db_html 625 626That'll generate a detailed cover report in the "cover_db_html" 627directory, which you can then copy to a webserver, or inspect locally 628in a browser. 629 630Smoke testing 631------------- 632 633The Git test suite has support for smoke testing. Smoke testing is 634when you submit the results of a test run to a central server for 635analysis and aggregation. 636 637Running a smoke tester is an easy and valuable way of contributing to 638Git development, particularly if you have access to an uncommon OS on 639obscure hardware. 640 641After building Git you can generate a smoke report like this in the 642"t" directory: 643 644 make clean smoke 645 646You can also pass arguments via the environment. This should make it 647faster: 648 649 GIT_TEST_OPTS='--root=/dev/shm' TEST_JOBS=10 make clean smoke 650 651The "smoke" target will run the Git test suite with Perl's 652"TAP::Harness" module, and package up the results in a .tar.gz archive 653with "TAP::Harness::Archive". The former is included with Perl v5.10.1 654or later, but you'll need to install the latter from the CPAN. See the 655"Test coverage" section above for how you might do that. 656 657Once the "smoke" target finishes you'll see a message like this: 658 659 TAP Archive created at <path to git>/t/test-results/git-smoke.tar.gz 660 661To upload the smoke report you need to have curl(1) installed, then 662do: 663 664 make smoke_report 665 666To upload the report anonymously. Hopefully that'll return something 667like "Reported #7 added.". 668 669If you're going to be uploading reports frequently please request a 670user account by E-Mailing gitsmoke@v.nix.is. Once you have a username 671and password you'll be able to do: 672 673 SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report 674 675You can also add an additional comment to attach to the report, and/or 676a comma separated list of tags: 677 678 SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> \ 679 SMOKE_COMMENT=<comment> SMOKE_TAGS=<tags> \ 680 make smoke_report 681 682Once the report is uploaded it'll be made available at 683http://smoke.git.nix.is, here's an overview of Recent Smoke Reports 684for Git: 685 686 http://smoke.git.nix.is/app/projects/smoke_reports/1 687 688The reports will also be mirrored to GitHub every few hours: 689 690 http://github.com/gitsmoke/smoke-reports 691 692The Smolder SQLite database is also mirrored and made available for 693download: 694 695 http://github.com/gitsmoke/smoke-database 696 697Note that the database includes hashed (with crypt()) user passwords 698and E-Mail addresses. Don't use a valuable password for the smoke 699service if you have an account, or an E-Mail address you don't want to 700be publicly known. The user accounts are just meant to be convenient 701labels, they're not meant to be secure.