When an object is not found after checking the packfiles and then
loose object directory, read_sha1_file() re-checks the packfiles to
prevent racing with a concurrent repacker; teach the same logic to
has_sha1_file().
* jk/has-sha1-file-retry-packed:
has_sha1_file: re-check pack directory before giving up
Earlier we started rejecting an attempt to add 0{40} object name to
the index and to tree objects, but it sometimes is necessary to
allow so to be able to use tools like filter-branch to correct such
broken tree objects.
* kk/tests-with-no-perl:
reset test: modernize style
t/t7106-reset-unborn-branch.sh: Add PERL prerequisite
add -i test: use skip_all instead of repeated PERL prerequisite
Make test "using invalid commit with -C" more strict
"git commit --author=$name", when $name is not in the canonical
"A. U. Thor <au.thor@example.xz>" format, looks for a matching name
from existing history, but did not consult mailmap to grab the
preferred author name.
* ap/commit-author-mailmap:
commit: search author pattern against mailmap
* mm/mediawiki-dumb-push-fix:
git-remote-mediawiki: no need to update private ref in non-dumb push
git-remote-mediawiki: use no-private-update capability on dumb push
transport-helper: add no-private-update capability
git-remote-mediawiki: add test and check Makefile targets
"git config" did not provide a way to set or access numbers larger
than a native "int" on the platform; it now provides 64-bit signed
integers on all platforms.
* jk/config-int-range-check:
git-config: always treat --int as 64-bit internally
config: make numeric parsing errors more clear
config: set errno in numeric git_parse_* functions
config: properly range-check integer values
config: factor out integer parsing from range checks
"git ls-files -k" needs to crawl only the part of the working tree
that may overlap the paths in the index to find killed files, but
shared code with the logic to find all the untracked files, which
made it unnecessarily inefficient.
* jc/ls-files-killed-optim:
dir.c::test_one_path(): work around directory_exists_in_index_icase() breakage
t3010: update to demonstrate "ls-files -k" optimization pitfalls
ls-files -k: a directory only can be killed if the index has a non-directory
dir.c: use the cache_* macro to access the current index
The commit object names in the insn sheet that was prepared at the
beginning of "rebase -i" session can become ambiguous as the
rebasing progresses and the repository gains more commits. Make
sure the internal record is kept with full 40-hex object names.
* es/rebase-i-no-abbrev:
rebase -i: fix short SHA-1 collision
t3404: rebase -i: demonstrate short SHA-1 collision
t3404: make tests more self-contained
* tf/gitweb-ss-tweak:
gitweb: make search help link less ugly
gitweb: omit the repository owner when it is unset
gitweb: vertically centre contents of page footer
gitweb: ensure OPML text fits inside its box
Update post-receive-email script to make sure the message contents
and pathnames are encoded consistently in UTF-8.
* jn/post-receive-utf8:
hooks/post-receive-email: set declared encoding to utf-8
hooks/post-receive-email: force log messages in UTF-8
hooks/post-receive-email: use plumbing instead of git log/show
Use "struct pathspec" interface in more places, instead of array of
characters, the latter of which cannot express magic pathspecs
(e.g. ":(icase)makefile" that matches both Makefile and makefile).
* nd/magic-pathspec:
add: lift the pathspec magic restriction on "add -p"
pathspec: catch prepending :(prefix) on pathspec with short magic
Send a large request to read(2)/write(2) as a smaller but still
reasonably large chunks, which would improve the latency when the
operation needs to be killed and incidentally works around broken
64-bit systems that cannot take a 2GB write or read in one go.
* sp/clip-read-write-to-8mb:
Revert "compat/clipped-write.c: large write(2) fails on Mac OS X/XNU"
xread, xwrite: limit size of IO to 8MB
The auto-tag-following code in "git fetch" tries to reuse the same
transport twice when the serving end does not cooperate and does
not give tags that point to commits that are asked for as part of
the primary transfer. Unfortunately, Git-aware transport helper
interface is not designed to be used more than once, hence this
does not work over smart-http transfer.
* jc/transport-do-not-use-connect-twice-in-fetch:
builtin/fetch.c: Fix a sparse warning
fetch: work around "transport-take-over" hack
fetch: refactor code that fetches leftover tags
fetch: refactor code that prepares a transport
fetch: rename file-scope global "transport" to "gtransport"
t5802: add test for connect helper
Allow section.<urlpattern>.var configuration variables to be
treated as a "virtual" section.var given a URL, and use the
mechanism to enhance http.* configuration variables.
This is a reroll of Kyle J. McKay's work.
* jc/url-match:
builtin/config.c: compilation fix
config: "git config --get-urlmatch" parses section.<url>.key
builtin/config: refactor collect_config()
config: parse http.<url>.<variable> using urlmatch
config: add generic callback wrapper to parse section.<url>.key
config: add helper to normalize and match URLs
http.c: fix parsing of http.sslCertPasswordProtected variable
"git mv A B" when moving a submodule A does "the right thing",
inclusing relocating its working tree and adjusting the paths in
the .gitmodules file.
* jl/submodule-mv: (53 commits)
rm: delete .gitmodules entry of submodules removed from the work tree
mv: update the path entry in .gitmodules for moved submodules
submodule.c: add .gitmodules staging helper functions
mv: move submodules using a gitfile
mv: move submodules together with their work trees
rm: do not set a variable twice without intermediate reading.
t6131 - skip tests if on case-insensitive file system
parse_pathspec: accept :(icase)path syntax
pathspec: support :(glob) syntax
pathspec: make --literal-pathspecs disable pathspec magic
pathspec: support :(literal) syntax for noglob pathspec
kill limit_pathspec_to_literal() as it's only used by parse_pathspec()
parse_pathspec: preserve prefix length via PATHSPEC_PREFIX_ORIGIN
parse_pathspec: make sure the prefix part is wildcard-free
rename field "raw" to "_raw" in struct pathspec
tree-diff: remove the use of pathspec's raw[] in follow-rename codepath
remove match_pathspec() in favor of match_pathspec_depth()
remove init_pathspec() in favor of parse_pathspec()
remove diff_tree_{setup,release}_paths
convert common_prefix() to use struct pathspec
...
Teaches "git blame" to take more than one -L ranges.
* es/blame-L-twice:
line-range: reject -L line numbers less than 1
t8001/t8002: blame: add tests of -L line numbers less than 1
line-range: teach -L^:RE to search from start of file
line-range: teach -L:RE to search from end of previous -L range
line-range: teach -L^/RE/ to search from start of file
line-range-format.txt: document -L/RE/ relative search
log: teach -L/RE/ to search from end of previous -L range
blame: teach -L/RE/ to search from end of previous -L range
line-range: teach -L/RE/ to search relative to anchor point
blame: document multiple -L support
t8001/t8002: blame: add tests of multiple -L options
blame: accept multiple -L ranges
blame: inline one-line function into its lone caller
range-set: publish API for re-use by git-blame -L
line-range-format.txt: clarify -L:regex usage form
git-log.txt: place each -L option variation on its own line
Output from "git log --full-diff -- <pathspec>" looked strange,
because comparison was done with the previous ancestor that touched
the specified <pathspec>, causing the patches for paths outside the
pathspec to show more than the single commit has changed.
Tweak "git reflog -p" for the same reason using the same mechanism.
* tr/log-full-diff-keep-true-parents:
log: use true parents for diff when walking reflogs
log: use true parents for diff even when rewriting
Allow a safer "rewind of the remote tip" push than blind "--force",
by requiring that the overwritten remote ref to be unchanged since
the new history to replace it was prepared.
The machinery is more or less ready. The "--force" option is again
the big red button to override any safety, thanks to J6t's sanity
(the original round allowed --lockref to defeat --force).
The logic to choose the default implemented here is fragile
(e.g. "git fetch" after seeing a failure will update the
remote-tracking branch and will make the next "push" pass,
defeating the safety pretty easily). It is suitable only for the
simplest workflows, and it may hurt users more than it helps them.
* jc/push-cas:
push: teach --force-with-lease to smart-http transport
send-pack: fix parsing of --force-with-lease option
t5540/5541: smart-http does not support "--force-with-lease"
t5533: test "push --force-with-lease"
push --force-with-lease: tie it all together
push --force-with-lease: implement logic to populate old_sha1_expect[]
remote.c: add command line option parser for "--force-with-lease"
builtin/push.c: use OPT_BOOL, not OPT_BOOLEAN
cache.h: move remote/connect API out of it
Teach "git diff --diff-filter" to express "I do not want to see
these classes of changes" more directly by listing only the
unwanted ones in lowercase (e.g. "--diff-filter=d" will show
everything but deletion) and deprecate "diff-files -q" which did
the same thing as "--diff-filter=d".
* jc/diff-filter-negation:
diff: deprecate -q option to diff-files
diff: allow lowercase letter to specify what change class to exclude
diff: reject unknown change class given to --diff-filter
diff: preparse --diff-filter string argument
diff: factor out match_filter()
diff: pass the whole diff_options to diffcore_apply_filter()
Allow fetch.prune and remote.*.prune configuration variables to be set,
and "git fetch" to behave as if "--prune" is given.
"git fetch" that honors remote.*.prune is fine, but I wonder if we
should somehow make "git push" aware of it as well. Perhaps
remote.*.prune should not be just a boolean, but a 4-way "none",
"push", "fetch", "both"?
* ms/fetch-prune-configuration:
fetch: make --prune configurable
git-config: always treat --int as 64-bit internally
When you run "git config --int", the maximum size of integer
you get depends on how git was compiled, and what it
considers to be an "int".
This is almost useful, because your scripts calling "git
config" will behave similarly to git internally. But relying
on this is dubious; you have to actually know how git treats
each value internally (e.g., int versus unsigned long),
which is not documented and is subject to change. And even
if you know it is "unsigned long", we do not have a
git-config option to match that behavior.
Furthermore, you may simply be asking git to store a value
on your behalf (e.g., configuration for a hook). In that
case, the relevant range check has nothing at all to do with
git, but rather with whatever scripting tools you are using
(and git has no way of knowing what the appropriate range is
there).
Not only is the range check useless, but it is actively
harmful, as there is no way at all for scripts to look
at config variables with large values. For instance, one
cannot reliably get the value of pack.packSizeLimit via
git-config. On an LP64 system, git happily uses a 64-bit
"unsigned long" internally to represent the value, but the
script cannot read any value over 2G.
Ideally, the "--int" option would simply represent an
arbitrarily large integer. For practical purposes, however,
a 64-bit integer is large enough, and is much easier to
implement (and if somebody overflows it, we will still
notice the problem, and not simply return garbage).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If we try to parse an integer config argument and get a
number outside of the representable range, we die with the
cryptic message: "bad config value for '%s'".
We can improve two things:
1. Show the value that produced the error (e.g., bad
config value '3g' for 'foo.bar').
2. Mention the reason the value was rejected (e.g.,
"invalid unit" versus "out of range").
A few tests need to be updated with the new output, but that
should not be representative of real-world breakage, as
scripts should not be depending on the exact text of our
stderr output, which is subject to i18n anyway.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: set errno in numeric git_parse_* functions
When we are parsing an integer or unsigned long, we use
the strto*max functions, which properly set errno to ERANGE
if we get a large value. However, we also do further range
checks after applying our multiplication factor, but do not
set ERANGE. This means that a caller cannot tell if an error
was caused by ERANGE or if the input was simply not a valid
number.
This patch teaches git_parse_signed and git_parse_unsigned to set
ERANGE for range errors, and EINVAL for other errors, so that the
caller can reliably tell these cases apart.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we look at a config value as an integer using the
git_config_int function, we carefully range-check the value
we get and complain if it is out of our range. But the range
we compare to is that of a "long", which we then cast to an
"int" in the function's return value. This means that on
systems where "int" and "long" have different sizes (e.g.,
LP64 systems), we may pass the range check, but then return
nonsense by truncating the value as we cast it to an int.
We can solve this by converting git_parse_long into
git_parse_int, and range-checking the "int" range. Nobody
actually cared that we used a "long" internally, since the
result was truncated anyway. And the only other caller of
git_parse_long is git_config_maybe_bool, which should be
fine to just use int (though we will now forbid out-of-range
nonsense like setting "merge.ff" to "10g" to mean "true",
which is probably a good thing).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
config: factor out integer parsing from range checks
When we are parsing integers for config, we use an intmax_t
(or uintmax_t) internally, and then check against the size
of our result type at the end. We can parameterize the
maximum representable value, which will let us re-use the
parsing code for a variety of range checks.
Unfortunately, we cannot combine the signed and unsigned
parsing functions easily, as we have to rely on the signed
and unsigned C types internally.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Pass a list of open bzrlib.transport.Transport objects to each bzrlib
function that might create a transport. This enables bzrlib to reuse
existing transports when possible, avoiding multiple concurrent
connections to the same remote server.
If the remote server is accessed via ssh, this fixes a couple of
problems:
* If the user does not have keys loaded into an ssh agent, the user
may be prompted for a password multiple times.
* If the user is using OpenSSH and the ControlMaster setting is set
to auto, git-remote-bzr might hang. This is because bzrlib closes
the multiple ssh sessions in an undefined order and might try to
close the master ssh session before the other sessions. The
master ssh process will not exit until the other sessions have
exited, causing a deadlock. (The ssh sessions are closed in an
undefined order because bzrlib relies on the Python garbage
collector to trigger ssh session termination.)
Signed-off-by: Richard Hansen <rhansen@bbn.com> Acked-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
add: lift the pathspec magic restriction on "add -p"
Since 480ca64 (convert run_add_interactive to use struct pathspec -
2013-07-14), we have unconditionally passed :(prefix)xxx to
add-interactive.perl. It implies that all commands
add-interactive.perl calls must be aware of pathspec magic, or
:(prefix) is barfed. The restriction to :/ only becomes unnecessary.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
pathspec: catch prepending :(prefix) on pathspec with short magic
:(prefix) is in the long form. Suppose people pass :!foo with '!'
being the short form of magic 'bar', the code will happily turn it to
:(prefix..)!foo, which makes '!' part of the path and no longer a magic.
The correct form must be ':(prefix..,bar)foo', but as so far we
haven't had any magic in short form yet (*), the code to convert from
short form to long one will be inactive anyway. Let's postpone it
until a real short form magic appears.
(*) The short form magic '/' is a special case and won't be caught by
this die(), which is correct. When '/' magic is detected, prefixlen is
set back to 0 and the whole "if (prefixlen..)" block is skipped.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a user is working on master, and has merged in their feature branch, but now
has to "git pull" because master moved, with pull.rebase their feature branch
will be flattened into master.
This is because "git pull" currently does not know about rebase's preserve
merges flag, which would avoid this behavior, as it would instead replay just
the merge commit of the feature branch onto the new master, and not replay each
individual commit in the feature branch.
Add a --rebase=preserve option, which will pass along --preserve-merges to
rebase.
Also add 'preserve' to the allowed values for the pull.rebase config setting.
Signed-off-by: Stephen Haberman <stephen@exigencecorp.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Convert most uses of OPT_BOOLEAN/OPTION_BOOLEAN that can use
OPT_BOOL/OPTION_BOOLEAN which have much saner semantics, and turn
remaining ones into OPT_SET_INT, OPT_COUNTUP, etc. as necessary.
* sb/parseopt-boolean-removal:
revert: use the OPT_CMDMODE for parsing, reducing code
checkout-index: fix negations of even numbers of -n
config parsing options: allow one flag multiple times
hash-object: replace stdin parsing OPT_BOOLEAN by OPT_COUNTUP
branch, commit, name-rev: ease up boolean conditions
checkout: remove superfluous local variable
log, format-patch: parsing uses OPT__QUIET
Replace deprecated OPT_BOOLEAN by OPT_BOOL
Remove deprecated OPTION_BOOLEAN for parsing arguments
Many commands use --dashed-option as a operation mode selector
(e.g. "git tag --delete") that the user can use at most one
(e.g. "git tag --delete --verify" is a nonsense) and you cannot
negate (e.g. "git tag --no-delete" is a nonsense). Make it easier
for users of parse_options() to enforce these restrictions.
* jc/parseopt-command-modes:
tag: use OPT_CMDMODE
parse-options: add OPT_CMDMODE()
Fixes a minor bug in "git rebase -i" (there could be others, as the
root cause is pretty generic) where the code feeds a random, data
dependeant string to 'echo' and expects it to come out literally.
* mm/no-shell-escape-in-die-message:
die_with_status: use "printf '%s\n'", not "echo"
* da/darwin:
OS X: Fix redeclaration of die warning
Makefile: Fix APPLE_COMMON_CRYPTO with BLK_SHA1
imap-send: use Apple's Security framework for base64 encoding
* jk/fast-import-empty-ls:
fast-import: allow moving the root tree
fast-import: allow ls or filecopy of the root tree
fast-import: set valid mode on root tree in "ls"
t9300: document fast-import empty path issues
Subversion 1.8.0 that was recently released breaks older subversion
clients coming over http/https in various ways.
* km/svn-1.8-serf-only:
Git.pm: revert _temp_cache use of temp_is_locked
git-svn: allow git-svn fetching to work using serf
Git.pm: add new temp_is_locked function
"git check-ignore -z" applied the NUL termination to both its input
(with --stdin) and its output, but "git check-attr -z" ignored the
option on the output side.
This is potentially a backward incompatible fix. Let's see if
anybody screams before deciding if we want to do anything to help
existing users (there may be none).
* jc/check-x-z:
check-attr -z: a single -z should apply to both input and output
check-ignore -z: a single -z should apply to both input and output
check-attr: the name of the character is NUL, not NULL
check-ignore: the name of the character is NUL, not NULL
git-remote-mediawiki: no need to update private ref in non-dumb push
We used to update the private ref ourselves, but this update is now
done by default since 664059fb (transport-helper: update remote
helper namespace, 2013-04-17).
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since 664059fb (transport-helper: update remote helper namespace,
2013-04-17), a 'push' operation on a remote helper updates the
private ref by default. This is often a good thing, but it can also
be desirable to disable this update to force the next 'pull' to
re-import the pushed revisions.
Allow remote-helpers to disable the automatic update by introducing a new
capability.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr> Signed-off-by: Junio C Hamano <gitster@pobox.com>
gitweb: Fix the author initials in blame for non-ASCII names
Change the @author_initials feature Jakub added in v1.6.4-rc2-14-ga36817b to match non-ASCII author initials as intended.
The regexp Jakub added was intended to match
non-ASCII (/\b([[:upper:]])\B/g). But in Perl this doesn't actually
match non-ASCII upper-case characters unless the string being matched
against has the UTF8 flag.
So when we open a pipe to "git blame" we need to mark the file
descriptor we're opening as utf8 explicitly.
So as a result it abbreviates me to "AB" not "ÆAB", entirely because "Æ"
isn't /[[:upper:]]/ unless the string being matched against has the UTF8
flag.
Here's something that demonstrates the issue:
#!/usr/bin/env perl
use strict;
use warnings;
binmode STDOUT, ':utf8' if $ENV{UTF8};
open my $fd, "-|", "git", "blame", "--incremental", "--", "Makefile" or die "Can't open: $!";
binmode $fd, ":utf8" if $ENV{UTF8};
while (my $line = <$fd>) {
next unless my ($author) = $line =~ /^author (.*)/;
my @author_initials = ($author =~ /\b([[:upper:]])\B/g);
printf "%s (%s)\n", join("", @author_initials), $author;
}
When that's run with and without UTF8 being true in the environment it
gives, on git.git:
Acked-by: Jakub Narębski <jnareb@gmail.com> Tested-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Tested-by: Simon Ruderich <simon@ruderich.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
has_sha1_file: re-check pack directory before giving up
When we read a sha1 file, we first look for a packed
version, then a loose version, and then re-check the pack
directory again before concluding that we cannot find it.
This lets us handle a process that is writing to the
repository simultaneously (e.g., receive-pack writing a new
pack followed by a ref update, or git-repack packing
existing loose objects into a new pack).
However, we do not do the same trick with has_sha1_file; we
only check the packed objects once, followed by loose
objects. This means that we might incorrectly report that we
do not have an object, even though we could find it if we
simply re-checked the pack directory.
By itself, this is usually not a big deal. The other process
is running simultaneously, so we may run has_sha1_file
before it writes, anyway. It is a race whether we see the
object or not. However, we may also see other things
the writing process has done (like updating refs); and in
that case, we must be able to also see the new objects.
For example, imagine we are doing a for_each_ref iteration,
and somebody simultaneously pushes. Receive-pack may write
the pack and update a ref after we have examined the
objects/pack directory, but before the iteration gets to the
updated ref. When we do finally see the updated ref,
for_each_ref will call has_sha1_file to check whether the
ref is broken. If has_sha1_file returns the wrong answer, we
erroneously will think that the ref is broken.
For a normal iteration without DO_FOR_EACH_INCLUDE_BROKEN,
this means that the caller does not see the ref at all
(neither the old nor the new value). So not only will we
fail to see the new value of the ref (which is acceptable,
since we are running simultaneously with the writer, and we
might well read the ref before the writer commits its
write), but we will not see the old value either. For
programs that act on reachability like pack-objects or
prune, this can cause data loss, as we may see the objects
referenced by the original ref value as dangling (and either
omit them from the pack, or delete them via prune).
There's no test included here, because the success case is
two processes running simultaneously forever. But you can
replicate the issue with:
# base.sh
# run this in one terminal; it creates and pushes
# repeatedly to a repository
git init parent &&
(cd parent &&
# create a base commit that will trigger us looking at
# the objects/pack directory before we hit the updated ref
echo content >file &&
git add file &&
git commit -m base &&
# set the unpack limit abnormally low, which
# lets us simulate full-size pushes using tiny ones
git config receive.unpackLimit 1
) &&
git clone parent child &&
cd child &&
n=0 &&
while true; do
echo $n >file && git add file && git commit -m $n &&
git push origin HEAD:refs/remotes/child/master &&
n=$(($n + 1))
done
# fsck.sh
# now run this simultaneously in another terminal; it
# repeatedly fscks, looking for us to consider the
# newly-pushed ref broken. We cannot use for-each-ref
# here, as it uses DO_FOR_EACH_INCLUDE_BROKEN, which
# skips the has_sha1_file check (and if it wants
# more information on the object, it will actually read
# the object, which does the proper two-step lookup)
cd parent &&
while true; do
broken=`git fsck 2>&1 | grep remotes/child`
if test -n "$broken"; then
echo $broken
exit 1
fi
done
Without this patch, the fsck loop fails within a few seconds
(and almost instantly if the test repository actually has a
large number of refs). With it, the two can run
indefinitely.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
It is tentatively called 1.8.5, but it should be an easy matter of
renaming the release-notes file and RelNotes symlink to later call
it 1.9 near the end of the cycle if we wanted to.
Some people still use rather old versions of bash, which cannot
grok some constructs like 'printf -v varname' the prompt and
completion code started to use recently.
* bc/completion-for-bash-3.0:
contrib/git-prompt.sh: handle missing 'printf -v' more gracefully
t9902-completion.sh: old Bash still does not support array+=('') notation
git-completion.bash: use correct Bash/Zsh array length syntax