1Use of index and Racy git problem 2================================= 3 4Background 5---------- 6 7The index is one of the most important data structure in git. 8It represents a virtual working tree state by recording list of 9paths and their object names and serves as a staging area to 10write out the next tree object to be committed. The state is 11"virtual" in the sense that it does not necessarily have to, and 12often does not, match the files in the working tree. 13 14There are cases git needs to examine the differences between the 15virtual working tree state in the index and the files in the 16working tree. The most obvious case is when the user asks `git 17diff` (or its low level implementation, `git diff-files`) or 18`git-ls-files --modified`. In addition, git internally checks 19if the files in the working tree is different from what are 20recorded in the index to avoid stomping on local changes in them 21during patch application, switching branches, and merging. 22 23In order to speed up this comparison between the files in the 24working tree and the index entries, the index entries record the 25information obtained from the filesystem via `lstat(2)` system 26call when they were last updated. When checking if they differ, 27git first runs `lstat(2)` on the files and compare the result 28with this information (this is what was originally done by the 29`ce_match_stat()` function, which the current code does in 30`ce_match_stat_basic()` function). If some of these "cached 31stat information" fields do not match, git can tell that the 32files are modified without even looking at their contents. 33 34Note: not all members in `struct stat` obtained via `lstat(2)` 35are used for this comparison. For example, `st_atime` obviously 36is not useful. Currently, git compares the file type (regular 37files vs symbolic links) and executable bits (only for regular 38files) from `st_mode` member, `st_mtime` and `st_ctime` 39timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members. 40With a `USE_STDEV` compile-time option, `st_dev` is also 41compared, but this is not enabled by default because this member 42is not stable on network filesystems. With `USE_NSEC` 43compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec` 44members are also compared, but this is not enabled by default 45because the value of this member becomes meaningless once the 46inode is evicted from the inode cache on filesystems that do not 47store it on disk. 48 49 50Racy git 51-------- 52 53There is one slight problem with the optimization based on the 54cached stat information. Consider this sequence: 55 56 $ git update-index 'foo' 57 : modify 'foo' in-place without changing its size 58 59The first `update-index` computes the object name of the 60contents of file `foo` and updates the index entry for `foo` 61along with the `struct stat` information. If the modification 62that follows it happens very fast so that the file's `st_mtime` 63timestamp does not change, after this sequence, the cached stat 64information the index entry records still exactly match what you 65can obtain from the filesystem, but the file `foo` is modified. 66This way, git can incorrectly think files in the working tree 67are unmodified even though they actually are. This is called 68the "racy git" problem (discovered by Pasky), and the entries 69that appear clean when they may not be because of this problem 70are called "racily clean". 71 72To avoid this problem, git does two things: 73 74. When the cached stat information says the file has not been 75 modified, and the `st_mtime` is the same as (or newer than) 76 the timestamp of the index file itself (which is the time `git 77 update-index foo` finished running in the above example), it 78 also compares the contents with the object registered in the 79 index entry to make sure they match. 80 81. When the index file is updated that contains racily clean 82 entries, cached `st_size` information is truncated to zero 83 before writing a new version of the index file. 84 85Because the index file itself is written after collecting all 86the stat information from updated paths, `st_mtime` timestamp of 87it is usually the same as or newer than any of the paths the 88index contains. And no matter how quick the modification that 89follows `git update-index foo` finishes, the resulting 90`st_mtime` timestamp on `foo` cannot get the timestamp earlier 91than the index file. Therefore, index entries that can be 92racily clean are limited to the ones that have the same 93timestamp as the index file itself. 94 95The callers that want to check if an index entry matches the 96corresponding file in the working tree continue to call 97`ce_match_stat()`, but with this change, `ce_match_stat()` uses 98`ce_modified_check_fs()` to see if racily clean ones are 99actually clean after comparing the cached stat information using 100`ce_match_stat_basic()`. 101 102The problem the latter solves is this sequence: 103 104 $ git update-index 'foo' 105 : modify 'foo' in-place without changing its size 106 : wait for enough time 107 $ git update-index 'bar' 108 109Without the latter, the timestamp of the index file gets a newer 110value, and falsely clean entry `foo` would not be caught by the 111timestamp comparison check done with the former logic anymore. 112The latter makes sure that the cached stat information for `foo` 113would never match with the file in the working tree, so later 114checks by `ce_match_stat_basic()` would report the index entry 115does not match the file and git does not have to fall back on more 116expensive `ce_modified_check_fs()`. 117 118 119Runtime penalty 120--------------- 121 122The runtime penalty of falling back to `ce_modified_check_fs()` 123from `ce_match_stat()` can be very expensive when there are many 124racily clean entries. An obvious way to artificially create 125this situation is to give the same timestamp to all the files in 126the working tree in a large project, run `git update-index` on 127them, and give the same timestamp to the index file: 128 129 $ date >.datestamp 130 $ git ls-files | xargs touch -r .datestamp 131 $ git ls-files | git update-index --stdin 132 $ touch -r .datestamp .git/index 133 134This will make all index entries racily clean. The linux-2.6 135project, for example, there are over 20,000 files in the working 136tree. On my Athron 64X2 3800+, after the above: 137 138 $ /usr/bin/time git diff-files 139 1.68user 0.54system 0:02.22elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 140 0inputs+0outputs (0major+67111minor)pagefaults 0swaps 141 $ git update-index MAINTAINERS 142 $ /usr/bin/time git diff-files 143 0.02user 0.12system 0:00.14elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 144 0inputs+0outputs (0major+935minor)pagefaults 0swaps 145 146Running `git update-index` in the middle checked the racily 147clean entries, and left the cached `st_mtime` for all the paths 148intact because they were actually clean (so this step took about 149the same amount of time as the first `git diff-files`). After 150that, they are not racily clean anymore but are truly clean, so 151the second invocation of `git diff-files` fully took advantage 152of the cached stat information. 153 154 155Avoiding runtime penalty 156------------------------ 157 158In order to avoid the above runtime penalty, the recent "master" 159branch (post 1.4.2) has a code that makes sure the index file 160gets timestamp newer than the youngest files in the index when 161there are many young files with the same timestamp as the 162resulting index file would otherwise would have by waiting 163before finishing writing the index file out. 164 165I suspect that in practice the situation where many paths in the 166index are all racily clean is quite rare. The only code paths 167that can record recent timestamp for large number of paths I 168know of are: 169 170. Initial `git add .` of a large project. 171 172. `git checkout` of a large project from an empty index into an 173 unpopulated working tree. 174 175Note: switching branches with `git checkout` keeps the cached 176stat information of existing working tree files that are the 177same between the current branch and the new branch, which are 178all older than the resulting index file, and they will not 179become racily clean. Only the files that are actually checked 180out can become racily clean. 181 182In a large project where raciness avoidance cost really matters, 183however, the initial computation of all object names in the 184index takes more than one second, and the index file is written 185out after all that happens. Therefore the timestamp of the 186index file will be more than one seconds later than the the 187youngest file in the working tree. This means that in these 188cases there actually will not be any racily clean entry in 189the resulting index. 190 191So in summary I think we should not worry about avoiding the 192runtime penalty and get rid of the "wait before finishing 193writing" code out.