In Perl, setting $/ sets the string that is used as the "record
separator," which sets the boundary that the `<>` construct reads to.
Setting `local $/ = 0666;` evaluates the octal, getting 438, and
stringifies it. Thus, the later read from `<CHLD_OUT>` stops as soon
as it encounters the string "438" in the watchman output, yielding
invalid JSON; repositories containing filenames with SHA1 hashes are
able to trip this easily.
Set `$/` to undefined, thus slurping all output from watchman. Also
close STDIN which is provided to watchman, to better guarantee that we
cannot deadlock with watchman while both attempting to read.
Signed-off-by: Alex Vandiver <alexmv@dropbox.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Instead of just taking $ENV{'PWD'}, use the same logic that converts
PWD to $git_work_tree on MSYS_NT in the watchman integration hook
script also on MINGW.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add a test utility (test-drop-caches) that flushes all changes to disk
then drops file system cache on Windows, Linux, and OSX.
Add a perf test (p7519-fsmonitor.sh) for fsmonitor.
By default, the performance test will utilize the Watchman file system
monitor if it is installed. If Watchman is not installed, it will use a
dummy integration script that does not report any new or modified files.
The dummy script has very little overhead which provides optimistic results.
The performance test will also use the untracked cache feature if it is
available as fsmonitor uses it to speed up scanning for untracked files.
There are 4 environment variables that can be used to alter the default
behavior of the performance test:
GIT_PERF_7519_UNTRACKED_CACHE: used to configure core.untrackedCache
GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex
GIT_PERF_7519_FSMONITOR: used to configure core.fsmonitor
GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests
The big win for using fsmonitor is the elimination of the need to scan the
working directory looking for changed and untracked files. If the file
information is all cached in RAM, the benefits are reduced.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
fsmonitor: add a sample integration script for Watchman
This script integrates the new fsmonitor capabilities of git with the
cross platform Watchman file watching service. To use the script:
Download and install Watchman from https://facebook.github.io/watchman/.
Rename the sample integration hook from fsmonitor-watchman.sample to
fsmonitor-watchman. Configure git to use the extension:
Optionally turn on the untracked cache for optimal performance.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Christian Couder <christian.couder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Test the ability to add/remove the fsmonitor index extension via
update-index.
Test that dirty files returned from the integration script are properly
represented in the index extension and verify that ls-files correctly
reports their state.
Test that ensure status results are correct when using the new fsmonitor
extension. Test untracked, modified, and new files by ensuring the
results are identical to when not using the extension.
Test that if the fsmonitor extension doesn't tell git about a change, it
doesn't discover it on its own. This ensures git is honoring the
extension and that we get the performance benefits desired.
Three test integration scripts are provided:
fsmonitor-all - marks all files as dirty
fsmonitor-none - marks no files as dirty
fsmonitor-watchman - integrates with Watchman with debug logging
To run tests in the test suite while utilizing fsmonitor:
First copy t/t7519/fsmonitor-all to a location in your path and then set
GIT_FORCE_PRELOAD_TEST=true and GIT_FSMONITOR_TEST=fsmonitor-all and run
your tests.
Note: currently when using the test script fsmonitor-watchman on
Windows, many tests fail due to a reported but not yet fixed bug in
Watchman where it holds on to handles for directories and files which
prevents the test directory from being cleaned up properly.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
split-index: disable the fsmonitor extension when running the split index test
The split index test t1700-split-index.sh has hard coded SHA values for
the index. Currently it supports index V4 and V3 but assumes there are
no index extensions loaded.
When manually forcing the fsmonitor extension to be turned on when
running the test suite, the SHA values no longer match which causes the
test to fail.
The potential matrix of index extensions and index versions can is quite
large so instead temporarily disable the extension before attempting to
run the test until the underlying problem of hard coded SHA values is fixed.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
fsmonitor: teach git to optionally utilize a file system monitor to speed up detecting new or changed files.
When the index is read from disk, the fsmonitor index extension is used
to flag the last known potentially dirty index entries. The registered
core.fsmonitor command is called with the time the index was last
updated and returns the list of files changed since that time. This list
is used to flag any additional dirty cache entries and untracked cache
directories.
We can then use this valid state to speed up preload_index(),
ie_match_stat(), and refresh_cache_ent() as they do not need to lstat()
files to detect potential changes for those entries marked
CE_FSMONITOR_VALID.
In addition, if the untracked cache is turned on valid_cached_dir() can
skip checking directories for new or changed files as fsmonitor will
invalidate the cache only for those directories that have been
identified as having potential changes.
To keep the CE_FSMONITOR_VALID state accurate during git operations;
when git updates a cache entry to match the current state on disk,
it will now set the CE_FSMONITOR_VALID bit.
Inversely, anytime git changes a cache entry, the CE_FSMONITOR_VALID bit
is cleared and the corresponding untracked cache directory is marked
invalid.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
update-index: add a new --force-write-index option
At times, it makes sense to avoid the cost of writing out the index
when the only changes can easily be recomputed on demand. This causes
problems when trying to write test cases to verify that state as they
can't guarantee the state has been persisted to disk.
Add a new option (--force-write-index) to update-index that will
ensure the index is written out even if the cache_changed flag is not
set.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
preload-index: add override to enable testing preload-index
By default, the preload index code path doesn't run unless there is a
minimum of 1000 files. To enable running the test suite and having it
execute the preload-index path, add an environment variable
(GIT_FORCE_PRELOAD_TEST) which will override that minimum and set it to 2.
This enables you run existing tests and have the core.preloadindex code
path execute as long as the test has at least 2 files by setting
GIT_FORCE_PRELOAD_TEXT=1 before running the test.
Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
"git branch -M a b" while on a branch that is completely unrelated
to either branch a or branch b misbehaved when multiple worktree
was in use. This has been fixed.
* nd/worktree-kill-parse-ref:
branch: fix branch renaming not updating HEADs correctly
In addition to "cc: <a@dd.re.ss> # cruft", "cc: a@dd.re.ss # cruft"
was taught to "git send-email" as a valid way to tell it that it
needs to also send a carbon copy to <a@dd.re.ss> in the trailer
section.
* mm/send-email-cc-cruft:
send-email: don't use Mail::Address, even if available
send-email: fix garbage removal after address
Merge branch 'rs/archive-excluded-directory' into maint
"git archive" did not work well with pathspecs and the
export-ignore attribute.
We may want to resurrect the "we don't archive an empty directory"
bonus patch, but I do not mind merging the above early to 'next'
and leave it as a separate follow-up enhancement.
cf. <20170820090629.tumvqwzkromcykjf@sigill.intra.peff.net>
* rs/archive-excluded-directory:
archive: don't queue excluded directories
archive: factor out helper functions for handling attributes
t5001: add tests for export-ignore attributes and exclude pathspecs
Killing "git merge --edit" before the editor returns control left
the repository in a state with MERGE_MSG but without MERGE_HEAD,
which incorrectly tells the subsequent "git commit" that there was
a squash merge in progress. This has been fixed.
* mg/killed-merge:
merge: save merge state earlier
merge: split write_merge_state in two
merge: clarify call chain
Documentation/git-merge: explain --continue
"git apply" that is used as a better "patch -p1" failed to apply a
taken from a file with CRLF line endings to a file with CRLF line
endings. The root cause was because it misused convert_to_git()
that tried to do "safe-crlf" processing by looking at the index
entry at the same path, which is a nonsense---in that mode, "apply"
is not working on the data in (or derived from) the index at all.
This has been fixed.
* tb/apply-with-crlf:
apply: file commited with CRLF should roundtrip diff and apply
convert: add SAFE_CRLF_KEEP_CRLF
Merge branch 'cc/subprocess-handshake-missing-capabilities' into maint
When handshake with a subprocess filter notices that the process
asked for an unknown capability, Git did not report what program
the offending subprocess was running. This has been corrected.
We may want a follow-up fix to tighten the error checking, though.
* cc/subprocess-handshake-missing-capabilities:
sub-process: print the cmd when a capability is unsupported
"git svn" used with "--localtime" option did not compute the tz
offset for the timestamp in question and instead always used the
current time, which has been corrected.
* ur/svn-local-zone:
git svn fetch: Create correct commit timestamp when using --localtime
"git am -s" has been taught that some input may end with a trailer
block that is not Signed-off-by: and it should refrain from adding
an extra blank line before adding a new sign-off in such a case.
* pw/am-signoff:
am: fix signoff when other trailers are present
Merge branch 'ma/pager-per-subcommand-action' into maint
The "tag.pager" configuration variable was useless for those who
actually create tag objects, as it interfered with the use of an
editor. A new mechanism has been introduced for commands to enable
pager depending on what operation is being carried out to fix this,
and then "git tag -l" is made to run pager by default.
If this works out OK, I think there are low-hanging fruits in
other commands like "git branch" that outputs long list in one mode
while taking input in another.
* ma/pager-per-subcommand-action:
git.c: ignore pager.* when launching builtin as dashed external
tag: change default of `pager.tag` to "on"
tag: respect `pager.tag` in list-mode only
t7006: add tests for how git tag paginates
git.c: provide setup_auto_pager()
git.c: let builtins opt for handling `pager.foo` themselves
builtin.h: take over documentation from api-builtin.txt
"git log --tag=no-such-tag" showed log starting from HEAD, which
has been fixed---it now shows nothing.
* jk/rev-list-empty-input:
revision: do not fallback to default when rev_input_given is set
rev-list: don't show usage when we see empty ref patterns
revision: add rev_input_given flag
t6018: flesh out empty input/output rev-list tests
Merge branch 'st/lib-gpg-kill-stray-agent' into maint
Some versions of GnuPG fails to kill gpg-agent it auto-spawned
and such a left-over agent can interfere with a test. Work it
around by attempting to kill one before starting a new test.
* st/lib-gpg-kill-stray-agent:
t: lib-gpg: flush gpg agent on startup
Conversion from uchar[20] to struct object_id continues; this is to
ensure that we do not assume sizeof(struct object_id) is the same
as the length of SHA-1 hash (or length of longest hash we support).
* po/read-graft-line:
commit: rewrite read_graft_line
commit: allocate array using object_id size
commit: replace the raw buffer with strbuf in read_graft_line
sha1_file: fix definition of null_sha1
"branch --set-upstream" that has been deprecated in Git 1.8 has
finally been retired.
* ks/branch-set-upstream:
branch: quote branch/ref names to improve readability
builtin/branch: stop supporting the "--set-upstream" option
t3200: cleanup cruft of a test
Killing "git merge --edit" before the editor returns control left
the repository in a state with MERGE_MSG but without MERGE_HEAD,
which incorrectly tells the subsequent "git commit" that there was
a squash merge in progress. This has been fixed.
* mg/killed-merge:
merge: save merge state earlier
merge: split write_merge_state in two
merge: clarify call chain
Documentation/git-merge: explain --continue
The code to acquire a lock on a reference (e.g. while accepting a
push from a client) used to immediately fail when the reference is
already locked---now it waits for a very short while and retries,
which can make it succeed if the lock holder was holding it during
a read-only operation.
* mh/ref-lock-entry:
refs: retry acquiring reference locks for 100ms
"[gc] rerereResolved = 5.days" used to be invalid, as the variable
is defined to take an integer counting the number of days. It now
is allowed.
* jc/cutoff-config:
rerere: allow approxidate in gc.rerereResolved/gc.rerereUnresolved
rerere: represent time duration in timestamp_t internally
t4200: parameterize "rerere gc" custom expiry test
t4200: gather "rerere gc" together
t4200: make "rerere gc" test more robust
t4200: give us a clean slate after "rerere gc" tests
We used to spend more than necessary cycles allocating and freeing
piece of memory while writing each index entry out. This has been
optimized.
* kw/write-index-reduce-alloc:
read-cache: avoid allocating every ondisk entry when writing
read-cache: fix memory leak in do_write_index
perf: add test for writing the index
"git apply" that is used as a better "patch -p1" failed to apply a
taken from a file with CRLF line endings to a file with CRLF line
endings. The root cause was because it misused convert_to_git()
that tried to do "safe-crlf" processing by looking at the index
entry at the same path, which is a nonsense---in that mode, "apply"
is not working on the data in (or derived from) the index at all.
This has been fixed.
* tb/apply-with-crlf:
apply: file commited with CRLF should roundtrip diff and apply
convert: add SAFE_CRLF_KEEP_CRLF
Test update to improve coverage for "git stash" operations.
* jt/stash-tests:
stash: add a test for stashing in a detached state
stash: add a test for when apply fails during stash branch
stash: add a test for stash create with no files
"git interpret-trailers" has been taught a "--parse" and a few
other options to make it easier for scripts to grab existing
trailer lines from a commit log message.
* jk/trailers-parse:
doc/interpret-trailers: fix "the this" typo
pretty: support normalization options for %(trailers)
t4205: refactor %(trailers) tests
pretty: move trailer formatting to trailer.c
interpret-trailers: add --parse convenience option
interpret-trailers: add an option to unfold values
interpret-trailers: add an option to show only existing trailers
interpret-trailers: add an option to show only the trailers
trailer: put process_trailers() options into a struct
A handful of bugfixes and an improvement to "diff --color-moved".
* jt/diff-color-move-fix:
diff: define block by number of alphanumeric chars
diff: respect MIN_BLOCK_LENGTH for last block
diff: avoid redundantly clearing a flag
"git diff" has been taught to optionally paint new lines that are
the same as deleted lines elsewhere differently from genuinely new
lines.
* sb/diff-color-move: (25 commits)
diff: document the new --color-moved setting
diff.c: add dimming to moved line detection
diff.c: color moved lines differently, plain mode
diff.c: color moved lines differently
diff.c: buffer all output if asked to
diff.c: emit_diff_symbol learns about DIFF_SYMBOL_SUMMARY
diff.c: emit_diff_symbol learns about DIFF_SYMBOL_STAT_SEP
diff.c: convert word diffing to use emit_diff_symbol
diff.c: convert show_stats to use emit_diff_symbol
diff.c: convert emit_binary_diff_body to use emit_diff_symbol
submodule.c: migrate diff output to use emit_diff_symbol
diff.c: emit_diff_symbol learns DIFF_SYMBOL_REWRITE_DIFF
diff.c: emit_diff_symbol learns about DIFF_SYMBOL_BINARY_FILES
diff.c: emit_diff_symbol learns DIFF_SYMBOL_HEADER
diff.c: emit_diff_symbol learns DIFF_SYMBOL_FILEPAIR_{PLUS, MINUS}
diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_INCOMPLETE
diff.c: emit_diff_symbol learns DIFF_SYMBOL_WORDS[_PORCELAIN]
diff.c: migrate emit_line_checked to use emit_diff_symbol
diff.c: emit_diff_symbol learns DIFF_SYMBOL_NO_LF_EOF
diff.c: emit_diff_symbol learns DIFF_SYMBOL_CONTEXT_FRAGINFO
...
We check the date of epoch timestamp candidates already with
starts_with(). Move beyond that part using skip_prefix() instead of
checking it again using a regular expression. Also group the minutes
part, so that we can access them using a substring match instead of
using a magic number.
Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
apply: check date of potential epoch timestamps first
has_epoch_timestamp() looks for time stamps that amount to either
1969-12-31 24:00 or 1970-01-01 00:00 after applying the time zone
offset. Move the check for these two dates up, set the expected hour
based on which one is found, or exit early if none of them are present,
thus avoiding to engage the regex machinery for newer dates.
This also gets rid of two magic string length constants.
Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
sha1-lookup: remove sha1_entry_pos() from header file
Since f1068efefe (sha1_file: drop experimental GIT_USE_LOOKUP search, 2017-08-09)
the definition of sha1_entry_pos() has been removed from "sha1-lookup.c", so
there is no need anymore for its declaration in "sha1-lookup.h".
Signed-off-by: Christian Couder <chriscool@tuxfamily.org> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
send-email: don't use Mail::Address, even if available
Using Mail::Address made sense when we didn't have a proper parser. We
now have a reasonable address parser, and using Mail::Address
_if available_ causes much more trouble than it gives benefits:
* Developers typically test one version, not both.
* Users may not be aware that installing Mail::Address will change the
behavior. They may complain about the behavior in one case without
knowing that Mail::Address is involved.
* Having this optional Mail::Address makes it tempting to anwser "please
install Mail::Address" to users instead of fixing our own code. We've
reached the stage where bugs in our parser should be fixed, not worked
around.
Signed-off-by: Matthieu Moy <git@matthieu-moy.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is a followup over 9d33439 (send-email: only allow one address
per body tag, 2017-02-20). The first iteration did allow writting
Cc: <foo@example.com> # garbage
but did so by matching the regex ([^>]*>?), i.e. stop after the first
instance of '>'. However, it did not properly deal with
Cc: foo@example.com # garbage
Fix this using a new function strip_garbage_one_address, which does
essentially what the old ([^>]*>?) was doing, but dealing with more
corner-cases. Since we've allowed
Cc: "Foo # Bar" <foobar@example.com>
in previous versions, it makes sense to continue allowing it (but we
still remove any garbage after it). OTOH, when an address is given
without quoting, we just take the first word and ignore everything
after.
Signed-off-by: Matthieu Moy <git@matthieu-moy.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
branch: fix branch renaming not updating HEADs correctly
There are two bugs that sort of work together and cause
problems. Let's start with one in replace_each_worktree_head_symref.
Before fa099d2322 (worktree.c: kill parse_ref() in favor of
refs_resolve_ref_unsafe() - 2017-04-24), this code looks like this:
if (strcmp(oldref, worktrees[i]->head_ref))
continue;
set_worktree_head_symref(...);
After fa099d2322, it is possible that head_ref can be NULL. However,
the updated code takes the wrong exit. In the error case (NULL
head_ref), we should "continue;" to the next worktree. The updated
code makes us _skip_ "continue;" and update HEAD anyway.
The NULL head_ref is triggered by the second bug in add_head_info (in
the same commit). With the flag RESOLVE_REF_READING, resolve_ref_unsafe()
will abort if it cannot resolve the target ref. For orphan checkouts,
HEAD always points to an unborned branch, resolving target ref will
always fail. Now we have NULL head_ref. Now we always update HEAD.
Correct the logic in replace_ function so that we don't accidentally
update HEAD on error. As it turns out, correcting the logic bug above
breaks branch renaming completely, thanks to the second bug.
"git branch -[Mm]" does two steps (on a normal checkout, no orphan!):
- rename the branch on disk (e.g. refs/heads/abc to refs/heads/def)
- update HEAD if it points to the branch being renamed.
At the second step, since the branch pointed to by HEAD (e.g. "abc") no
longer exists on disk, we run into a temporary orphan checkout situation
that has been just corrected to _not_ update HEAD. But we need to update
HEAD since it's not actually an orphan checkout. We need to update HEAD
to move out of that orphan state.
Correct add_head_info(), remove RESOLVE_REF_READING flag. With the flag
gone, we should always return good "head_ref" in orphan checkouts (either
temporary or permanent). With good head_ref, things start to work again.
Noticed-by: Nish Aravamudan <nish.aravamudan@canonical.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
convert: display progress for filtered objects that have been delayed
In 2841e8f ("convert: add "status=delayed" to filter process protocol",
2017-06-30) we taught the filter process protocol to delayed responses.
These responses are processed after the "Checking out files" phase.
If the processing takes noticeable time, then the user might think Git
is stuck.
Display the progress of the delayed responses to let the user know that
Git is still processing objects. This works very well for objects that
can be filtered quickly. If filtering of an individual object takes
noticeable time, then the user might still think that Git is stuck.
However, in that case the user would at least know what Git is doing.
It would be technical more correct to display "Checking out files whose
content filtering has been delayed". For brevity we only print
"Filtering content".
The finish_delayed_checkout() call was moved below the stop_progress()
call in unpack-trees.c to ensure that the "Checking out files" progress
is properly stopped before the "Filtering content" progress starts in
finish_delayed_checkout().
Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Suggested-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Updates to the HTTP layer we made recently unconditionally used
features of libCurl without checking the existence of them, causing
compilation errors, which has been fixed. Also migrate the code to
check feature macros, not version numbers, to cope better with
libCurl that vendor ships with backported features.
* tc/curl-with-backports:
http: use a feature check to enable GSSAPI delegation control
http: fix handling of missing CURLPROTO_*
When handshake with a subprocess filter notices that the process
asked for an unknown capability, Git did not report what program
the offending subprocess was running. This has been corrected.
* cc/subprocess-handshake-missing-capabilities:
sub-process: print the cmd when a capability is unsupported