1Git performance tests 2===================== 3 4This directory holds performance testing scripts for git tools. The 5first part of this document describes the various ways in which you 6can run them. 7 8When fixing the tools or adding enhancements, you are strongly 9encouraged to add tests in this directory to cover what you are 10trying to fix or enhance. The later part of this short document 11describes how your test scripts should be organized. 12 13 14Running Tests 15------------- 16 17The easiest way to run tests is to say "make". This runs all 18the tests on the current git repository. 19 20 === Running 2 tests in this tree === 21 [...] 22 Test this tree 23 --------------------------------------------------------- 24 0001.1: rev-list --all 0.54(0.51+0.02) 25 0001.2: rev-list --all --objects 6.14(5.99+0.11) 26 7810.1: grep worktree, cheap regex 0.16(0.16+0.35) 27 7810.2: grep worktree, expensive regex 7.90(29.75+0.37) 28 7810.3: grep --cached, cheap regex 3.07(3.02+0.25) 29 7810.4: grep --cached, expensive regex 9.39(30.57+0.24) 30 31You can compare multiple repositories and even git revisions with the 32'run' script: 33 34 $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh 35 36where . stands for the current git tree. The full invocation is 37 38 ./run [<revision|directory>...] [--] [<test-script>...] 39 40A '.' argument is implied if you do not pass any other 41revisions/directories. 42 43You can also manually test this or another git build tree, and then 44call the aggregation script to summarize the results: 45 46 $ ./p0001-rev-list.sh 47 [...] 48 $ ./run /path/to/other/git -- ./p0001-rev-list.sh 49 [...] 50 $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh 51 52aggregate.perl has the same invocation as 'run', it just does not run 53anything beforehand. 54 55You can set the following variables (also in your config.mak): 56 57 GIT_PERF_REPEAT_COUNT 58 Number of times a test should be repeated for best-of-N 59 measurements. Defaults to 3. 60 61 GIT_PERF_MAKE_OPTS 62 Options to use when automatically building a git tree for 63 performance testing. E.g., -j6 would be useful. Passed 64 directly to make as "make $GIT_PERF_MAKE_OPTS". 65 66 GIT_PERF_MAKE_COMMAND 67 An arbitrary command that'll be run in place of the make 68 command, if set the GIT_PERF_MAKE_OPTS variable is 69 ignored. Useful in cases where source tree changes might 70 require issuing a different make command to different 71 revisions. 72 73 This can be (ab)used to monkeypatch or otherwise change the 74 tree about to be built. Note that the build directory can be 75 re-used for subsequent runs so the make command might get 76 executed multiple times on the same tree, but don't count on 77 any of that, that's an implementation detail that might change 78 in the future. 79 80 GIT_PERF_REPO 81 GIT_PERF_LARGE_REPO 82 Repositories to copy for the performance tests. The normal 83 repo should be at least git.git size. The large repo should 84 probably be about linux.git size for optimal results. 85 Both default to the git.git you are running from. 86 87You can also pass the options taken by ordinary git tests; the most 88useful one is: 89 90--root=<directory>:: 91 Create "trash" directories used to store all temporary data during 92 testing under <directory>, instead of the t/ directory. 93 Using this option with a RAM-based filesystem (such as tmpfs) 94 can massively speed up the test suite. 95 96 97Naming Tests 98------------ 99 100The performance test files are named as: 101 102 pNNNN-commandname-details.sh 103 104where N is a decimal digit. The same conventions for choosing NNNN as 105for normal tests apply. 106 107 108Writing Tests 109------------- 110 111The perf script starts much like a normal test script, except it 112sources perf-lib.sh: 113 114 #!/bin/sh 115 # 116 # Copyright (c) 2005 Junio C Hamano 117 # 118 119 test_description='xxx performance test' 120 . ./perf-lib.sh 121 122After that you will want to use some of the following: 123 124 test_perf_fresh_repo # sets up an empty repository 125 test_perf_default_repo # sets up a "normal" repository 126 test_perf_large_repo # sets up a "large" repository 127 128 test_perf_default_repo sub # ditto, in a subdir "sub" 129 130 test_checkout_worktree # if you need the worktree too 131 132At least one of the first two is required! 133 134You can use test_expect_success as usual. In both test_expect_success 135and in test_perf, running "git" points to the version that is being 136perf-tested. The $MODERN_GIT variable points to the git wrapper for the 137currently checked-out version (i.e., the one that matches the t/perf 138scripts you are running). This is useful if your setup uses commands 139that only work with newer versions of git than what you might want to 140test (but obviously your new commands must still create a state that can 141be used by the older version of git you are testing). 142 143For actual performance tests, use 144 145 test_perf 'descriptive string' ' 146 command1 && 147 command2 148 ' 149 150test_perf spawns a subshell, for lack of better options. This means 151that 152 153* you _must_ export all variables that you need in the subshell 154 155* you _must_ flag all variables that you want to persist from the 156 subshell with 'test_export': 157 158 test_perf 'descriptive string' ' 159 foo=$(git rev-parse HEAD) && 160 test_export foo 161 ' 162 163 The so-exported variables are automatically marked for export in the 164 shell executing the perf test. For your convenience, test_export is 165 the same as export in the main shell. 166 167 This feature relies on a bit of magic using 'set' and 'source'. 168 While we have tried to make sure that it can cope with embedded 169 whitespace and other special characters, it will not work with 170 multi-line data. 171 172Rather than tracking the performance by run-time as `test_perf` does, you 173may also track output size by using `test_size`. The stdout of the 174function should be a single numeric value, which will be captured and 175shown in the aggregated output. For example: 176 177 test_perf 'time foo' ' 178 ./foo >foo.out 179 ' 180 181 test_size 'output size' 182 wc -c <foo.out 183 ' 184 185might produce output like: 186 187 Test origin HEAD 188 ------------------------------------------------------------- 189 1234.1 time foo 0.37(0.79+0.02) 0.26(0.51+0.02) -29.7% 190 1234.2 output size 4.3M 3.6M -14.7% 191 192The item being measured (and its units) is up to the test; the context 193and the test title should make it clear to the user whether bigger or 194smaller numbers are better. Unlike test_perf, the test code will only be 195run once, since output sizes tend to be more deterministic than timings.