1Core GIT Tests 2============== 3 4This directory holds many test scripts for core GIT tools. The 5first part of this short document describes how to run the tests 6and read their output. 7 8When fixing the tools or adding enhancements, you are strongly 9encouraged to add tests in this directory to cover what you are 10trying to fix or enhance. The later part of this short document 11describes how your test scripts should be organized. 12 13 14Running Tests 15------------- 16 17The easiest way to run tests is to say "make". This runs all 18the tests. 19 20 *** t0000-basic.sh *** 21 ok 1 - .git/objects should be empty after git init in an empty repo. 22 ok 2 - .git/objects should have 3 subdirectories. 23 ok 3 - success is reported like this 24 ... 25 ok 43 - very long name in the index handled sanely 26 # fixed 1 known breakage(s) 27 # still have 1 known breakage(s) 28 # passed all remaining 42 test(s) 29 1..43 30 *** t0001-init.sh *** 31 ok 1 - plain 32 ok 2 - plain with GIT_WORK_TREE 33 ok 3 - plain bare 34 35Since the tests all output TAP (see http://testanything.org) they can 36be run with any TAP harness. Here's an example of paralell testing 37powered by a recent version of prove(1): 38 39 $ prove --timer --jobs 15 ./t[0-9]*.sh 40 [19:17:33] ./t0005-signals.sh ................................... ok 36 ms 41 [19:17:33] ./t0022-crlf-rename.sh ............................... ok 69 ms 42 [19:17:33] ./t0024-crlf-archive.sh .............................. ok 154 ms 43 [19:17:33] ./t0004-unwritable.sh ................................ ok 289 ms 44 [19:17:33] ./t0002-gitfile.sh ................................... ok 480 ms 45 ===( 102;0 25/? 6/? 5/? 16/? 1/? 4/? 2/? 1/? 3/? 1... )=== 46 47prove and other harnesses come with a lot of useful options. The 48--state option in particular is very useful: 49 50 # Repeat until no more failures 51 $ prove -j 15 --state=failed,save ./t[0-9]*.sh 52 53You can also run each test individually from command line, like this: 54 55 $ sh ./t3010-ls-files-killed-modified.sh 56 ok 1 - git update-index --add to add various paths. 57 ok 2 - git ls-files -k to show killed files. 58 ok 3 - validate git ls-files -k output. 59 ok 4 - git ls-files -m to show modified files. 60 ok 5 - validate git ls-files -m output. 61 # passed all 5 test(s) 62 1..5 63 64You can pass --verbose (or -v), --debug (or -d), and --immediate 65(or -i) command line argument to the test, or by setting GIT_TEST_OPTS 66appropriately before running "make". 67 68--verbose:: 69 This makes the test more verbose. Specifically, the 70 command being run and their output if any are also 71 output. 72 73--debug:: 74 This may help the person who is developing a new test. 75 It causes the command defined with test_debug to run. 76 77--immediate:: 78 This causes the test to immediately exit upon the first 79 failed test. 80 81--long-tests:: 82 This causes additional long-running tests to be run (where 83 available), for more exhaustive testing. 84 85--valgrind:: 86 Execute all Git binaries with valgrind and exit with status 87 126 on errors (just like regular tests, this will only stop 88 the test script when running under -i). Valgrind errors 89 go to stderr, so you might want to pass the -v option, too. 90 91 Since it makes no sense to run the tests with --valgrind and 92 not see any output, this option implies --verbose. For 93 convenience, it also implies --tee. 94 95--tee:: 96 In addition to printing the test output to the terminal, 97 write it to files named 't/test-results/$TEST_NAME.out'. 98 As the names depend on the tests' file names, it is safe to 99 run the tests with this option in parallel. 100 101--with-dashes:: 102 By default tests are run without dashed forms of 103 commands (like git-commit) in the PATH (it only uses 104 wrappers from ../bin-wrappers). Use this option to include 105 the build directory (..) in the PATH, which contains all 106 the dashed forms of commands. This option is currently 107 implied by other options like --valgrind and 108 GIT_TEST_INSTALLED. 109 110--root=<directory>:: 111 Create "trash" directories used to store all temporary data during 112 testing under <directory>, instead of the t/ directory. 113 Using this option with a RAM-based filesystem (such as tmpfs) 114 can massively speed up the test suite. 115 116You can also set the GIT_TEST_INSTALLED environment variable to 117the bindir of an existing git installation to test that installation. 118You still need to have built this git sandbox, from which various 119test-* support programs, templates, and perl libraries are used. 120If your installed git is incomplete, it will silently test parts of 121your built version instead. 122 123When using GIT_TEST_INSTALLED, you can also set GIT_TEST_EXEC_PATH to 124override the location of the dashed-form subcommands (what 125GIT_EXEC_PATH would be used for during normal operation). 126GIT_TEST_EXEC_PATH defaults to `$GIT_TEST_INSTALLED/git --exec-path`. 127 128 129Skipping Tests 130-------------- 131 132In some environments, certain tests have no way of succeeding 133due to platform limitation, such as lack of 'unzip' program, or 134filesystem that do not allow arbitrary sequence of non-NUL bytes 135as pathnames. 136 137You should be able to say something like 138 139 $ GIT_SKIP_TESTS=t9200.8 sh ./t9200-git-cvsexport-commit.sh 140 141and even: 142 143 $ GIT_SKIP_TESTS='t[0-4]??? t91?? t9200.8' make 144 145to omit such tests. The value of the environment variable is a 146SP separated list of patterns that tells which tests to skip, 147and either can match the "t[0-9]{4}" part to skip the whole 148test, or t[0-9]{4} followed by ".$number" to say which 149particular test to skip. 150 151Note that some tests in the existing test suite rely on previous 152test item, so you cannot arbitrarily disable one and expect the 153remainder of test to check what the test originally was intended 154to check. 155 156 157Naming Tests 158------------ 159 160The test files are named as: 161 162 tNNNN-commandname-details.sh 163 164where N is a decimal digit. 165 166First digit tells the family: 167 168 0 - the absolute basics and global stuff 169 1 - the basic commands concerning database 170 2 - the basic commands concerning the working tree 171 3 - the other basic commands (e.g. ls-files) 172 4 - the diff commands 173 5 - the pull and exporting commands 174 6 - the revision tree commands (even e.g. merge-base) 175 7 - the porcelainish commands concerning the working tree 176 8 - the porcelainish commands concerning forensics 177 9 - the git tools 178 179Second digit tells the particular command we are testing. 180 181Third digit (optionally) tells the particular switch or group of switches 182we are testing. 183 184If you create files under t/ directory (i.e. here) that is not 185the top-level test script, never name the file to match the above 186pattern. The Makefile here considers all such files as the 187top-level test script and tries to run all of them. A care is 188especially needed if you are creating a common test library 189file, similar to test-lib.sh, because such a library file may 190not be suitable for standalone execution. 191 192 193Writing Tests 194------------- 195 196The test script is written as a shell script. It should start 197with the standard "#!/bin/sh" with copyright notices, and an 198assignment to variable 'test_description', like this: 199 200 #!/bin/sh 201 # 202 # Copyright (c) 2005 Junio C Hamano 203 # 204 205 test_description='xxx test (option --frotz) 206 207 This test registers the following structure in the cache 208 and tries to run git-ls-files with option --frotz.' 209 210 211Source 'test-lib.sh' 212-------------------- 213 214After assigning test_description, the test script should source 215test-lib.sh like this: 216 217 . ./test-lib.sh 218 219This test harness library does the following things: 220 221 - If the script is invoked with command line argument --help 222 (or -h), it shows the test_description and exits. 223 224 - Creates an empty test directory with an empty .git/objects database 225 and chdir(2) into it. This directory is 't/trash 226 directory.$test_name_without_dotsh', with t/ subject to change by 227 the --root option documented above. 228 229 - Defines standard test helper functions for your scripts to 230 use. These functions are designed to make all scripts behave 231 consistently when command line arguments --verbose (or -v), 232 --debug (or -d), and --immediate (or -i) is given. 233 234 235End with test_done 236------------------ 237 238Your script will be a sequence of tests, using helper functions 239from the test harness library. At the end of the script, call 240'test_done'. 241 242 243Test harness library 244-------------------- 245 246There are a handful helper functions defined in the test harness 247library for your script to use. 248 249 - test_expect_success <message> <script> 250 251 This takes two strings as parameter, and evaluates the 252 <script>. If it yields success, test is considered 253 successful. <message> should state what it is testing. 254 255 Example: 256 257 test_expect_success \ 258 'git-write-tree should be able to write an empty tree.' \ 259 'tree=$(git-write-tree)' 260 261 - test_expect_failure <message> <script> 262 263 This is NOT the opposite of test_expect_success, but is used 264 to mark a test that demonstrates a known breakage. Unlike 265 the usual test_expect_success tests, which say "ok" on 266 success and "FAIL" on failure, this will say "FIXED" on 267 success and "still broken" on failure. Failures from these 268 tests won't cause -i (immediate) to stop. 269 270 - test_debug <script> 271 272 This takes a single argument, <script>, and evaluates it only 273 when the test script is started with --debug command line 274 argument. This is primarily meant for use during the 275 development of a new test script. 276 277 - test_done 278 279 Your test script must have test_done at the end. Its purpose 280 is to summarize successes and failures in the test script and 281 exit with an appropriate error code. 282 283 - test_tick 284 285 Make commit and tag names consistent by setting the author and 286 committer times to defined stated. Subsequent calls will 287 advance the times by a fixed amount. 288 289 - test_commit <message> [<filename> [<contents>]] 290 291 Creates a commit with the given message, committing the given 292 file with the given contents (default for both is to reuse the 293 message string), and adds a tag (again reusing the message 294 string as name). Calls test_tick to make the SHA-1s 295 reproducible. 296 297 - test_merge <message> <commit-or-tag> 298 299 Merges the given rev using the given message. Like test_commit, 300 creates a tag and calls test_tick before committing. 301 302Tips for Writing Tests 303---------------------- 304 305As with any programming projects, existing programs are the best 306source of the information. However, do _not_ emulate 307t0000-basic.sh when writing your tests. The test is special in 308that it tries to validate the very core of GIT. For example, it 309knows that there will be 256 subdirectories under .git/objects/, 310and it knows that the object ID of an empty tree is a certain 31140-byte string. This is deliberately done so in t0000-basic.sh 312because the things the very basic core test tries to achieve is 313to serve as a basis for people who are changing the GIT internal 314drastically. For these people, after making certain changes, 315not seeing failures from the basic test _is_ a failure. And 316such drastic changes to the core GIT that even changes these 317otherwise supposedly stable object IDs should be accompanied by 318an update to t0000-basic.sh. 319 320However, other tests that simply rely on basic parts of the core 321GIT working properly should not have that level of intimate 322knowledge of the core GIT internals. If all the test scripts 323hardcoded the object IDs like t0000-basic.sh does, that defeats 324the purpose of t0000-basic.sh, which is to isolate that level of 325validation in one place. Your test also ends up needing 326updating when such a change to the internal happens, so do _not_ 327do it and leave the low level of validation to t0000-basic.sh.