fetch, upload-pack: --deepen=N extends shallow boundary by N commits
In git-fetch, --depth argument is always relative with the latest
remote refs. This makes it a bit difficult to cover this use case,
where the user wants to make the shallow history, say 3 levels
deeper. It would work if remote refs have not moved yet, but nobody
can guarantee that, especially when that use case is performed a
couple months after the last clone or "git fetch --depth". Also,
modifying shallow boundary using --depth does not work well with
clones created by --since or --not.
This patch fixes that. A new argument --deepen=<N> will add <N> more (*)
parent commits to the current history regardless of where remote refs
are.
Have/Want negotiation is still respected. So if remote refs move, the
server will send two chunks: one between "have" and "want" and another
to extend shallow history. In theory, the client could send no "want"s
in order to get the second chunk only. But the protocol does not allow
that. Either you send no want lines, which means ls-remote; or you
have to send at least one want line that carries deep-relative to the
server..
The main work was done by Dongcan Jiang. I fixed it up here and there.
And of course all the bugs belong to me.
(*) We could even support --deepen=<N> where <N> is negative. In that
case we can cut some history from the shallow clone. This operation
(and --depth=<shorter depth>) does not require interaction with remote
side (and more complicated to implement as a result).
Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
upload-pack: support define shallow boundary by excluding revisions
This should allow the user to say "create a shallow clone of this branch
after version <some-tag>".
Short refs are accepted and expanded at the server side with expand_ref()
because we cannot expand (unknown) refs from the client side.
Like deepen-since, deepen-not cannot be used with deepen. But deepen-not
can be mixed with deepen-since. The result is exactly how you do the
command "git rev-list --since=... --not ref".
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This is basically dwim_ref() without @{} support. To be used on the
server side where we want to expand abbreviated to full ref names and
nothing else. The first user is "git clone/fetch --shallow-exclude".
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
upload-pack: add deepen-since to cut shallow repos based on time
This should allow the user to say "create a shallow clone containing the
work from last year" (once the client side is fixed up, of course).
In theory deepen-since and deepen (aka --depth) can be used together to
draw the shallow boundary (whether it's intersection or union is up to
discussion, but if rev-list is used, it's likely intersection). However,
because deepen goes with a custom commit walker, we can't mix the two
yet.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
shallow.c: implement a generic shallow boundary finder based on rev-list
Instead of a custom commit walker like get_shallow_commits(), this new
function uses rev-list to mark NOT_SHALLOW to all reachable commits,
except borders. The definition of reachable is to be defined by the
protocol later. This makes it more flexible to define shallow boundary.
The way we find border is paint all reachable commits NOT_SHALLOW. Any
of them that "touches" commits without NOT_SHALLOW flag are considered
shallow (e.g. zero parents via grafting mechanism). Shallow commits and
their true parents are all marked SHALLOW. Then NOT_SHALLOW is removed
from shallow commits at the end.
There is an interesting observation. With a generic walker, we can
produce all kinds of shallow cutting. In the following graph, every
commit but "x" is reachable. "b" is a parent of "a".
x -- a -- o
/ /
x -- c -- b -- o
After this function is run, "a" and "c" are both considered shallow
commits. After grafting occurs at the client side, what we see is
a -- o
/
c -- b -- o
Notice that because of grafting, "a" has zero parents, so "b" is no
longer a parent of "a".
This is unfortunate and may be solved in two ways. The first is change
the way shallow grafting works and keep "a -- b" connection if "b"
exists and always ends at shallow commits (iow, no loose ends). This is
hard to detect, or at least not cheap to do.
The second way is mark one "x" as shallow commit instead of "a" and
produce this graph at client side:
x -- a -- o
/ /
c -- b -- o
More commits, but simpler grafting rules.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch-pack: use a separate flag for fetch in deepening mode
The shallow repo could be deepened or shortened when then user gives
--depth. But in future that won't be the only way to deepen/shorten a
repo. Stop relying on args->depth in this mode. Future deepening
methods can simply set this flag on instead of updating all these if
expressions.
The new name "deepen" was chosen after the command to define shallow
boundary in pack protocol. New commands also follow this tradition.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
fetch-pack: use a common function for verbose printing
This reduces the number of "if (verbose)" which makes it a bit easier
to read imo. It also makes it easier to redirect all these printouts,
to a file for example.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
upload-pack: make check_non_tip() clean things up on error
On error check_non_tip() will die and not closing file descriptors is no
big deal. The next patch will split the majority of this function out
for reuse in other cases, where die() may not be the only outcome. Same
story for popping SIGPIPE out of the signal chain. So let's make sure we
clean things up properly first.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
upload-pack: move "unshallow" sending code out of deepen()
Also add some more comments in this code because it takes too long to
understand what it does (to me, who should be familiar enough to
understand this code well!)
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
After the last patch, "result" and "backup" are the same. "result" used
to move, but the movement is now contained in send_shallow(). Delete
this redundant variable.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
For now we can handle two types, string and boolean, in
set_helper_option(). Later on we'll add string_list support, which does
not fit well. The new function strbuf_set_helper_option() can be reused
for a separate function that handles string-list.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-svn has not supported GIT_SVN_NO_OPTIMIZE_COMMITS for
the "set-tree" sub-command in 9 years since commit 490f49ea5899
("git-svn: remove optimized commit stuff for set-tree").
So remove this target and TSVN variable to avoid confusion.
Helped-by: Kazutoshi Satoda <k_satoda@f2.dion.ne.jp> Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Paths that have been told the index about with "add -N" are not
quite yet in the index, but a few commands behaved as if they
already are in a harmful way.
* nd/ita-cleanup:
grep: make it clear i-t-a entries are ignored
add and use a convenience macro ce_intent_to_add()
blame: remove obsolete comment
Merge branch 'dw/mergetool-vim-window-shuffle' into maint
The vimdiff backend for "git mergetool" has been tweaked to arrange
and number buffers in the order that would match the expectation of
majority of people who read left to right, then top down and assign
buffers 1 2 3 4 "mentally" to local base remote merge windows based
on that order.
* dw/mergetool-vim-window-shuffle:
mergetool: reorder vim/gvim buffers in three-way diffs
Merge branch 'ks/svn-pathnameencoding-4' of git://git.bogomips.org/git-svn
* 'ks/svn-pathnameencoding-4' of git://git.bogomips.org/git-svn:
git-svn: apply "svn.pathnameencoding" before URL encoding
git-svn: enable "svn.pathnameencoding" on dcommit
git-svn: hoist out utf8 prep from t9129 to lib-git-svn
git-svn: apply "svn.pathnameencoding" before URL encoding
The conversion from "svn.pathnameencoding" to UTF-8 should be applied
first, and then URL encoding should be applied on the resulting UTF-8
path. The reversed order of these transforms (used before this fix)
makes non-UTF-8 URL which causes error from Subversion such as
"Filesystem has no item: '...' path not found" when sending a rename (or
a copy) from non-ASCII path.
[ew: t9115 test case added (requires SVN_HTTPD_PORT set to test),
squash LC_ALL=$a_utf8_locale export from Kazutoshi for Cygwin]
Signed-off-by: Kazutoshi SATODA <k_satoda@f2.dion.ne.jp> Signed-off-by: Eric Wong <normalperson@yhbt.net>
Without the initialization of $self->{pathnameencoding}, conversion in
repo_path() is always skipped as $self->{pathnameencoding} is undefined
even if "svn.pathnameencoding" is configured.
The lack of conversion results in mysterious failure of dcommit (e.g.
"Malformed XML") which happen only when a commit involves a change on
non-ASCII path.
[ew: add test case to t9115,
squash LC_ALL=$a_utf8_locale export from Kazutoshi for Cygwin]
Signed-off-by: Kazutoshi SATODA <k_satoda@f2.dion.ne.jp> Signed-off-by: Eric Wong <normalperson@yhbt.net>
The vimdiff backend for "git mergetool" has been tweaked to arrange
and number buffers in the order that would match the expectation of
majority of people who read left to right, then top down and assign
buffers 1 2 3 4 "mentally" to local base remote merge windows based
on that order.
* dw/mergetool-vim-window-shuffle:
mergetool: reorder vim/gvim buffers in three-way diffs
Some calls to strcpy(3) triggers a false warning from static
analysers that are less intelligent than humans, and reducing the
number of these false hits helps us notice real issues. A few
calls to strcpy(3) in "git rerere" that are already safe has been
rewritten to avoid false wanings.
* jk/rerere-xsnprintf:
rerere: replace strcpy with xsnprintf
Some calls to strcpy(3) triggers a false warning from static
analysers that are less intelligent than humans, and reducing the
number of these false hits helps us notice real issues. A few
calls to strcpy(3) in test-path-utils that are already safe has
been rewritten to avoid false wanings.
* jk/test-path-utils-xsnprintf:
test-path-utils: use xsnprintf in favor of strcpy
The "user.useConfigOnly" configuration variable can be used to
force the user to always set user.email & user.name configuration
variables, serving as a reminder for those who work on multiple
projects and do not want to put these in their $HOME/.gitconfig.
* da/user-useconfigonly:
ident: add user.useConfigOnly boolean for when ident shouldn't be guessed
fmt_ident: refactor strictness checks
The automatic typo correction applied to an alias was broken
with a recent change already in 'master'.
* nd/clear-gitenv-upon-use-of-alias:
restore_env(): free the saved environment variable once we are done
git: simplify environment save/restore logic
git: protect against unbalanced calls to {save,restore}_env()
git: remove an early return from save_env_before_alias()
An earlier adjustment of test mistakenly used write_script
to prepare a file whose exact content matters for the test;
reverting that part fixes the breakage for those who use
SHELL_PATH that is different from /bin/sh.
* mg/mingw-test-fix:
t9100: fix breakage when SHELL_PATH is not /bin/sh
Test scripts have been updated to remove assumptions that are not
portable between Git for POSIX and Git for Windows, or to skip ones
with expectations that are not satisfiable on Git for Windows.
* js/mingw-tests: (21 commits)
gitignore: ignore generated test-fake-ssh executable
mingw: do not bother to test funny file names
mingw: skip a test in t9130 that cannot pass on Windows
mingw: handle the missing POSIXPERM prereq in t9124
mingw: avoid illegal filename in t9118
mingw: mark t9100's test cases with appropriate prereqs
t0008: avoid absolute path
mingw: work around pwd issues in the tests
mingw: fix t9700's assumption about directory separators
mingw: skip test in t1508 that fails due to path conversion
tests: turn off git-daemon tests if FIFOs are not available
mingw: disable mkfifo-based tests
mingw: accomodate t0060-path-utils for MSYS2
mingw: fix t5601-clone.sh
mingw: let lstat() fail with errno == ENOTDIR when appropriate
mingw: try to delete target directory before renaming
mingw: prepare the TMPDIR environment variable for shell scripts
mingw: factor out Windows specific environment setup
Git.pm: stop assuming that absolute paths start with a slash
mingw: do not trust MSYS2's MinGW gettext.sh
...
It turns out "git clone" over rsync transport has been broken when
the source repository has packed references for a long time, and
nobody noticed nor complained about it.
* jk/drop-rsync-transport:
transport: drop support for git-over-rsync
Merge branch 'js/test-lib-windows-emulated-yes' into maint
The emulated "yes" command used in our test scripts has been
tweaked not to spend too much time generating unnecessary output
that is not used, to help those who test on Windows where it would
not stop until it fills the pipe buffer due to lack of SIGPIPE.
* js/test-lib-windows-emulated-yes:
test-lib: limit the output of the yes utility
Merge branch 'nd/do-not-move-worktree-manually' into maint
"git worktree" had a broken code that attempted to auto-fix
possible inconsistency that results from end-users moving a
worktree to different places without telling Git (the original
repository needs to maintain backpointers to its worktrees, but
"mv" run by end-users who are not familiar with that fact will
obviously not adjust them), which actually made things worse
when triggered.
The low-level merge machinery has been taught to use CRLF line
termination when inserting conflict markers to merged contents that
are themselves CRLF line-terminated.
* js/xmerge-marker-eol:
merge-file: ensure that conflict sections match eol style
merge-file: let conflict markers match end-of-line style of the context
t5505: 'remote add x y' should work when url.y.insteadOf = x
This is the test missing from fb86e32 (git remote: allow adding
remotes agreeing with url.<...>.insteadOf, 2014-12-23): we should
allow adding a remote with the URL when it agrees with the
url.<...>.insteadOf setting.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Now that git_connect is more information about connectivity
progress after: ("pass transport verbosity down to git_connect")
we should ensure it remains so for future users who need to
to diagnose networking problems.
Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-config: better document default behavior for `--include`
As described in the commit message of 9b25a0b (config: add
include directive, 2012-02-06), the `--include` option is
only on by default in some cases. But our documentation
described it as just "defaults to on", which doesn't tell
the whole story.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
mergetool: reorder vim/gvim buffers in three-way diffs
When invoking default (g)vimdiff three-way merge, the merged file is
loaded as the first buffer but moved to the bottom as the fourth window.
This causes a disconnect between vim commands that operate on window
positions (e.g. CTRL-W_w) and those that operate on buffer index (e.g.
do/dp).
This change reorders the buffers to have the same index as windows while
keeping the cursor default to the merged result as the bottom window.
Signed-off-by: Dickson Wong <dicksonwong@gmail.com> Tested-by: Michael J Gruber <git@drmicha.warpmail.net> Acked-by: David Aguilar <davvid@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The emulated "yes" command used in our test scripts has been
tweaked not to spend too much time generating unnecessary output
that is not used, to help those who test on Windows where it would
not stop until it fills the pipe buffer due to lack of SIGPIPE.
* js/test-lib-windows-emulated-yes:
test-lib: limit the output of the yes utility
Clean/smudge filters defined in a configuration file of lower
precedence can now be overridden to be a pass-through no-op by
setting the variable to an empty string.
* ls/clean-smudge-override-in-config:
convert: treat an empty string for clean/smudge filters as "cat"
There were a few "now I am doing this thing" progress messages in
the TCP connection code that can be triggered by setting a verbose
option internally in the code, but "git fetch -v" and friends never
passed the verbose option down to that codepath.
There was a brief discussion about the impact on the end-user
experience by not limiting this to "fetch -v -v", but I think the
conclusion is that this is OK to enable with a single "-v" as it is
not too noisy.
* ew/connect-verbose:
pass transport verbosity down to git_connect
The low-level merge machinery has been taught to use CRLF line
termination when inserting conflict markers to merged contents that
are themselves CRLF line-terminated.
* js/xmerge-marker-eol:
merge-file: ensure that conflict sections match eol style
merge-file: let conflict markers match end-of-line style of the context
"git worktree" had a broken code that attempted to auto-fix
possible inconsistency that results from end-users moving a
worktree to different places without telling Git (the original
repository needs to maintain backpointers to its worktrees, but
"mv" run by end-users who are not familiar with that fact will
obviously not adjust them), which actually made things worse
when triggered.
Documentation/git-clean.txt: don't mention deletion of .git/modules/*
The latter half of this sentence, the removal of the submodules, was
never done with (or without) double -f back when it was written, and
we still do not do so.
Signed-off-by: Matt McCutchen <matt@mattmccutchen.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This shouldn't overflow, as we are copying a sha1 hex into a
41-byte buffer. But it does not hurt to use a bound-checking
function, which protects us and makes auditing for overflows
easier.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
t9100: fix breakage when SHELL_PATH is not /bin/sh
bcb11f1 (mingw: mark t9100's test cases with appropriate prereqs, 2016-01-27)
replaced "/bin/sh" in exec.sh by the shell specified in SHELL_PATH, but
that breaks the subtest which checks for a specific checksum of a tree
containing.
Revert that change that was not explained in the commit message anyways
(exec.sh is never executed).
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This strcpy will never overflow because it's copying from
baked-in test data. But we would prefer to avoid strcpy
entirely, as it makes it harder to audit for real security
bugs.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In "mingw: fix t5601-clone.sh", this developer introduced a new test
executable, test-fake-ssh but forgot to update the .gitignore file
accordingly. Fix that.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
ident: add user.useConfigOnly boolean for when ident shouldn't be guessed
It used to be that:
git config --global user.email "(none)"
was a viable way for people to force themselves to set user.email in
each repository. This was helpful for people with more than one
email address, targeting different email addresses for different
clones, as it barred git from creating a commit unless the user.email
config was set in the per-repo config to the correct email address.
A recent change, 19ce497c (ident: keep a flag for bogus
default_email, 2015-12-10), however, declared that an explicitly
configured user.email is not bogus, no matter what its value is, so
this hack no longer works.
Provide the same functionality by adding a new configuration
variable user.useConfigOnly; when this variable is set, the
user must explicitly set user.email configuration.
Signed-off-by: Junio C Hamano <gitster@pobox.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Dan Aloni <alonid@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'ss/clone-depth-single-doc' into maint
Documentation for "git fetch --depth" has been updated for clarity.
* ss/clone-depth-single-doc:
docs: clarify that --depth for git-fetch works with newly initialized repos
docs: say "commits" in the --depth option wording for git-clone
docs: clarify that passing --depth to git-clone implies --single-branch
Drop a few old "todo" items by deciding that the change one of them
suggests is not such a good idea, and doing the change the other
one suggested to do.
* ss/user-manual:
user-manual: add addition gitweb information
user-manual: add section documenting shallow clones
glossary: define the term shallow clone
user-manual: remove temporary branch entry from todo list
Merge branch 'jk/ref-cache-non-repository-optim' into maint
The underlying machinery used by "ls-files -o" and other commands
have been taught not to create empty submodule ref cache for a
directory that is not a submodule. This removes a ton of wasted
CPU cycles.
* jk/ref-cache-non-repository-optim:
resolve_gitlink_ref: ignore non-repository paths
clean: make is_git_repository a public function
dirname() emulation has been added, as Msys2 lacks it.
* js/dirname-basename:
mingw: avoid linking to the C library's isalpha()
t0060: loosen overly strict expectations
t0060: verify that basename() and dirname() work as expected
compat/basename.c: provide a dirname() compatibility function
compat/basename: make basename() conform to POSIX
Refactor skipping DOS drive prefixes
Merge branch 'jk/list-tag-2.7-regression' into maint
"git tag" started listing a tag "foo" as "tags/foo" when a branch
named "foo" exists in the same repository; remove this unnecessary
disambiguation, which is a regression introduced in v2.7.0.
* jk/list-tag-2.7-regression:
tag: do not show ambiguous tag names as "tags/foo"
t6300: use test_atom for some un-modern tests