commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily

The optional 'Extra Edge List' chunk of the commit graph file stores
parent information for commits with more than two parents. Since the
chunk is optional, write_commit_graph() looks through all commits to
find those with more than two parents, and then writes the commit
graph file header accordingly, i.e. if there are no such commits, then
there won't be a 'Extra Edge List' chunk written, only the three
mandatory chunks.

However, when it later comes to writing actual chunk data,
write_commit_graph() unconditionally invokes
write_graph_chunk_extra_edges(), even when it was decided earlier that
that chunk won't be written. Strictly speaking there is no bug here,
because write_graph_chunk_extra_edges() won't write anything if it
doesn't find any commits with more than two parents, but then it
unnecessarily and in vain looks through all commits once again in
search for such commits.

Don't call write_graph_chunk_extra_edges() when that chunk won't be
written to spare an unnecessary iteration over all commits.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: tidy asciidoc style

This mainly refers to enforcing indentation on additional lines of
items of lists.

Signed-off-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Add 'human' date format documentation

Display date and time information in a format similar to how people
write dates in other contexts. If the year isn't specified then, the
reader infers the date is given is in the current year.

By not displaying the redundant information, the reader concentrates
on the information that is different. The patch reports relative dates
based on information inferred from the date on the machine running the
git command at the time the command is executed.

While the format is more useful to humans by dropping inferred
information, there is nothing that makes it actually human. If the
'relative' date format wasn't already implemented then using
'relative' would have been appropriate.

Signed-off-by: Stephen P. Smith <ischis2@cox.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Replace the proposed 'auto' mode with 'auto:'

In addition to adding the 'human' format, the patch added the auto
keyword which could be used in the config file as an alternate way to
specify the human format. Removing 'auto' cleans up the 'human'
format interface.

Added the ability to specify mode 'foo' if the pager is being used by
using auto:foo syntax. Therefore, 'auto:human' date mode defaults to
human if we're using the pager. So you can do

git config --add log.date auto:human

and your "git log" commands will show the human-legible format unless
you're scripting things.

Signed-off-by: Stephen P. Smith <ischis2@cox.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

attr: do not mark queried macros as unset

Since 60a12722ac (attr: remove maybe-real, maybe-macro from git_attr,
2017-01-27), we will always mark an attribute macro (e.g., "binary")
that is specifically queried for as "unspecified", even though listing
_all_ attributes would display it at set. E.g.:

$ echo "* binary" >.gitattributes

$ git check-attr -a file
file: binary: set
file: diff: unset
file: merge: unset
file: text: unset

$ git check-attr binary file
file: binary: unspecified

The problem stems from an incorrect conversion of the optimization from
06a604e670 (attr: avoid heavy work when we know the specified attr is
not defined, 2014-12-28). There we tried in collect_some_attrs() to
avoid even looking at the attr_stack when the user has asked for "foo"
and we know that "foo" did not ever appear in any .gitattributes file.

It used a flag "maybe_real" in each attribute struct, where "real" meant
that the attribute appeared in an actual file (we have to make this
distinction because we also create an attribute struct for any names
that are being queried). But as explained in that commit message, the
meaning of "real" was tangled with some special cases around macros.

When 60a12722ac later refactored the macro code, it dropped maybe_real
entirely. This missed the fact that "maybe_real" could be unset for two
reasons: because of a macro, or because it was never found during
parsing. This had two results:

- the optimization in collect_some_attrs() ceased doing anything
meaningful, since it no longer kept track of "was it found during
parsing"

- worse, it actually kicked in when the caller _did_ ask about a macro
by name, causing us to mark it as unspecified

It should be possible to salvage this optimization, but let's start with
just removing the remnants. It hasn't been doing anything (except
creating bugs) since 60a12722ac, and nobody seems to have noticed the
performance regression. It's more important to fix the correctness
problem clearly first.

I've added two tests here. The second one actually shows off the bug.
The test of "check-attr -a" is not strictly necessary, but we currently
do not test attribute macros much, and the builtin "binary" not at all.
So this increases our general test coverage, as well as making sure we
didn't mess up this related case.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tests: explicitly use `test-tool.exe` on Windows

In 8abfdf44c882 (tests: explicitly use `git.exe` on Windows,
2018-11-14), we made sure to use the `.exe` file extension when
using an absolute path to `git.exe`, to avoid getting confused with a
file or directory in the same place that lacks said file extension.

For the same reason, we need to handle test-tool.exe the same way.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph: rename "large edges" to "extra edges"

The optional 'Large Edge List' chunk of the commit graph file stores
parent information for commits with more than two parents, and the
names of most of the macros, variables, struct fields, and functions
related to this chunk contain the term "large edges", e.g.
write_graph_chunk_large_edges(). However, it's not a really great
term, as the edges to the second and subsequent parents stored in this
chunk are not any larger than the edges to the first and second
parents stored in the "main" 'Commit Data' chunk. It's the number of
edges, IOW number of parents, that is larger compared to non-merge and
"regular" two-parent merge commits. And indeed, two functions in
'commit-graph.c' have a local variable called 'num_extra_edges' that
refer to the same thing, and this "extra edges" term is much better at
describing these edges.

So let's rename all these references to "large edges" in macro,
variable, function, etc. names to "extra edges". There is a
GRAPH_OCTOPUS_EDGES_NEEDED macro as well; for the sake of consistency
rename it to GRAPH_EXTRA_EDGES_NEEDED.

We can do so safely without causing any incompatibility issues,
because the term "large edges" doesn't come up in the file format
itself in any form (the chunk's magic is {'E', 'D', 'G', 'E'}, there
is no 'L' in there), but only in the specification text. The string
"large edges", however, does come up in the output of 'git
commit-graph read' and in tests looking at its input, but that command
is explicitly documented as debugging aid, so we can change its output
and the affected tests safely.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph write: use pack order when finding commits

Slightly optimize the "commit-graph write" step by using
FOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit
[1] and [2] for the facility and a similar optimization for "cat-file".

On Linux it is around 5% slower to run:

echo 1 >/proc/sys/vm/drop_caches &&
cat .git/objects/pack/* >/dev/null &&
git cat-file --batch-all-objects --batch-check --unordered

Than the same thing with the "cat" omitted. This is as expected, since
we're iterating in pack order and the "cat" is extra work.

Before this change the opposite was true of "commit-graph write". We
were 6% faster if we first ran "cat" to efficiently populate the FS
cache for our sole big pack on linux.git, than if we had populated it
via for_each_object_in_pack(). Now we're 3% faster without the "cat"
instead.

My tests were done on an unloaded Linux 3.10 system with 10 runs for
each. Derrick Stolee did his own tests on Windows[3] showing a 2%
improvement with a high degree of accuracy.

1. 736eb88fdc ("for_each_packed_object: support iterating in
pack-order", 2018-08-10)

2. 0750bb5b51 ("cat-file: support "unordered" output for
--batch-all-objects", 2018-08-10)

3. https://public-inbox.org/git/f71fa868-25e8-a9c9-46a6-611b987f1a8f@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-tree: add missing --gpg-sign flag

Add --gpg-sign option in commit-tree, which was documented, but not
implemented, in 55ca3f99ae. Add tests for the --gpg-sign option.

Signed-off-by: Brandon Richardson <brandon1024.br@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t7510: invoke git as part of &&-chain

If `git commit-tree HEAD^{tree}` fails on us and produces no output on
stdout, we will substitute that empty string and execute `git tag
ninth-unsigned`, i.e., we will tag HEAD rather than a newly created
object. But we are lucky: we have a signature on HEAD, so we should
eventually fail the next test, where we verify that "ninth-unsigned" is
indeed unsigned.

We have a similar problem a few lines later. If `git commit-tree -S`
fails with no output, we will happily tag HEAD as "tenth-signed". Here,
we are not so lucky. The tag ends up on the same commit as
"eighth-signed-alt", and that's a signed commit, so t7510-signed-commit
will pass, despite `git commit-tree -S` failing.

Make these `git commit-tree` invocations a direct part of the &&-chain,
so that we can rely less on luck and set a better example for future
tests modeled after this one. Fix a 9/10 copy/paste error while at it.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Brandon Richardson <brandon1024.br@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-submodule: abort if core.worktree could not be set correctly

74d4731da1f (submodule--helper: replace connect-gitdir-workingtree by
ensure-core-worktree, 2018-08-13) forgot to exit the submodule operation
if the helper could not ensure that core.worktree is set correctly.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Do not print 'dangling' for cat-file in case of ambiguity

The return values -1 and -2 from get_oid could mean two different
things, depending on whether they were from an enum returned by
get_tree_entry_follow_symlinks, or from a different code path. This
caused 'dangling' to be printed from a git cat-file in the case of an
ambiguous (-2) result.

Unify the results of get_oid* and get_tree_entry_follow_symlinks to be
one common type, with unambiguous values.

Signed-off-by: David Turner <novalis@novalis.org>
Reported-by: Eric Wong <e@80x24.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Third batch after 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>

Merge branch 'js/gc-repack-close-before-remove'

"git gc" and "git repack" did not close the open packfiles that
they found unneeded before removing them, which didn't work on a
platform incapable of removing an open file. This has been
corrected.

* js/gc-repack-close-before-remove:
gc/repack: release packs when needed

Merge branch 'en/show-ref-doc-fix'

Doc update.

* en/show-ref-doc-fix:
git-show-ref.txt: fix order of flags

Merge branch 'ot/ref-filter-object-info'

The "--format=<placeholder>" option of for-each-ref, branch and tag
learned to show a few more traits of objects that can be learned by
the object_info API.

* ot/ref-filter-object-info:
ref-filter: give uintmax_t to format with %PRIuMAX
ref-filter: add docs for new options
ref-filter: add tests for deltabase
ref-filter: add deltabase option
ref-filter: add tests for objectsize:disk
ref-filter: add check for negative file size
ref-filter: add objectsize:disk option

Merge branch 'sg/stress-test'

Flaky tests can now be repeatedly run under load with the
"--stress" option.

* sg/stress-test:
test-lib: add the '--stress' option to run a test repeatedly under load
test-lib-functions: introduce the 'test_set_port' helper function
test-lib: set $TRASH_DIRECTORY earlier
test-lib: consolidate naming of test-results paths
test-lib: parse command line options earlier
test-lib: parse options in a for loop to keep $@ intact
test-lib: extract Bash version check for '-x' tracing
test-lib: translate SIGTERM and SIGHUP to an exit

Merge branch 'rs/sha1-file-close-mapped-file-on-error'

Code clean-up.

* rs/sha1-file-close-mapped-file-on-error:
sha1-file: close fd of empty file in map_sha1_file_1()

Merge branch 'rs/loose-object-cache-perffix'

The loose object cache used to optimize existence look-up has been
updated.

* rs/loose-object-cache-perffix:
object-store: retire odb_load_loose_cache()
object-store: use one oid_array per subdirectory for loose cache
object-store: factor out odb_clear_loose_cache()
object-store: factor out odb_loose_cache()

Merge branch 'po/git-p4-wo-login'

"git p4" update.

* po/git-p4-wo-login:
git-p4: fix problem when p4 login is not necessary

Merge branch 'mm/multimail-1.5'

Update "git multimail" from the upstream.

* mm/multimail-1.5:
git-multimail: update to release 1.5.0

Merge branch 'tg/t5570-drop-racy-test'

An inherently racy test that caused intermittent failures has been
removed.

* tg/t5570-drop-racy-test:
Revert "t/lib-git-daemon: record daemon log"
t5570: drop racy test

Merge branch 'jk/dev-build-format-security'

Earlier we added "-Wformat-security" to developer builds, assuming
that "-Wall" (which includes "-Wformat" which in turn is required
to use "-Wformat-security") is always in effect. This is not true
when config.mak.autogen is in use, unfortunately. This has been
fixed by unconditionally adding "-Wall" to developer builds.

* jk/dev-build-format-security:
config.mak.dev: add -Wall, primarily for -Wformat, to help autoconf users

Merge branch 'so/cherry-pick-always-allow-m1'

"git cherry-pick -m1" was forbidden when picking a non-merge
commit, even though there _is_ parent number 1 for such a commit.
This was done to avoid mistakes back when "cherry-pick" was about
picking a single commit, but is no longer useful with "cherry-pick"
that can pick a range of commits. Now the "-m$num" option is
allowed when picking any commit, as long as $num names an existing
parent of the commit.

Technically this is a backward incompatible change; hopefully
nobody is relying on the error-checking behaviour.

* so/cherry-pick-always-allow-m1:
t3506: validate '-m 1 -ff' is now accepted for non-merge commits
t3502: validate '-m 1' argument is now accepted for non-merge commits
cherry-pick: do not error on non-merge commits when '-m 1' is specified
t3510: stop using '-m 1' to force failure mid-sequence of cherry-picks

Merge branch 'nd/worktree-remove-with-uninitialized-submodules'

"git worktree remove" and "git worktree move" refused to work when
there is a submodule involved. This has been loosened to ignore
uninitialized submodules.

* nd/worktree-remove-with-uninitialized-submodules:
worktree: allow to (re)move worktrees with uninitialized submodules

Merge branch 'sg/test-bash-version-fix'

The test suite tried to see if it is run under bash, but the check
itself failed under some other implementations of shell (notably
under NetBSD). This has been corrected.

* sg/test-bash-version-fix:
test-lib: check Bash version for '-x' without using shell arrays

Merge branch 'rb/hpe'

Portability updates for the HPE NonStop platform.

* rb/hpe:
compat/regex/regcomp.c: define intptr_t and uintptr_t on NonStop
git-compat-util.h: add FLOSS headers for HPE NonStop
config.mak.uname: support for modern HPE NonStop config.
transport-helper: drop read/write errno checks
transport-helper: use xread instead of read

Merge branch 'ed/simplify-setup-git-dir'

Code simplification.

* ed/simplify-setup-git-dir:
Simplify handling of setup_git_directory_gently() failure cases.

Merge branch 'cy/zsh-completion-SP-in-path'

With zsh, "git cmd path<TAB>" was completed to "git cmd path name"
when the completed path has a special character like SP in it,
without any attempt to keep "path name" a single filename. This
has been fixed to complete it to "git cmd path\ name" just like
Bash completion does.

* cy/zsh-completion-SP-in-path:
completion: treat results of git ls-tree as file paths
zsh: complete unquoted paths with spaces correctly

Merge branch 'cy/completion-typofix'

Typofix.

* cy/completion-typofix:
completion: fix typo in git-completion.bash

Merge branch 'ew/ban-strncat'

The "strncat()" function is now among the banned functions.

* ew/ban-strncat:
banned.h: mark strncat() as banned

Merge branch 'ds/commit-graph-assert-missing-parents'

Tightening error checking in commit-graph writer.

* ds/commit-graph-assert-missing-parents:
commit-graph: writing missing parents is a BUG

Merge branch 'es/doc-worktree-guessremote-config'

Doc clarification.

* es/doc-worktree-guessremote-config:
doc/config: do a better job of introducing 'worktree.guessRemote'

Merge branch 'sb/submodule-unset-core-worktree-when-worktree-is-lost'

The core.worktree setting in a submodule repository should not be
pointing at a directory when the submodule loses its working tree
(e.g. getting deinit'ed), but the code did not properly maintain
this invariant.

* sb/submodule-unset-core-worktree-when-worktree-is-lost:
submodule deinit: unset core.worktree
submodule--helper: fix BUG message in ensure_core_worktree
submodule: unset core.worktree if no working tree is present
submodule update: add regression test with old style setups

Merge branch 'ma/asciidoctor'

Some of the documentation pages formatted incorrectly with
Asciidoctor, which have been fixed.

* ma/asciidoctor:
git-status.txt: render tables correctly under Asciidoctor
Documentation: do not nest open blocks
git-column.txt: fix section header

Merge branch 'jn/stripspace-wo-repository'

"git stripspace" should be usable outside a git repository, but
under the "-s" or "-c" mode, it didn't.

* jn/stripspace-wo-repository:
stripspace: allow -s/-c outside git repository

Merge branch 'sb/submodule-fetchjobs-default-to-one'

"git submodule update" ought to use a single job unless asked, but
by mistake used multiple jobs, which has been fixed.

* sb/submodule-fetchjobs-default-to-one:
submodule update: run at most one fetch job unless otherwise set

Merge branch 'la/quiltimport-keep-non-patch'

"git quiltimport" learned "--keep-non-patch" option.

* la/quiltimport-keep-non-patch:
git-quiltimport: add --keep-non-patch option

Merge branch 'nd/style-opening-brace'

Code clean-up.

* nd/style-opening-brace:
style: the opening '{' of a function is in a separate line

Merge branch 'ds/gc-doc-typofix'

Typofix.

* ds/gc-doc-typofix:
git-gc.txt: fix typo about gc.writeCommitGraph

mingw: special-case arguments to `sh`

The MSYS2 runtime does its best to emulate the command-line wildcard
expansion and de-quoting which would be performed by the calling Unix
shell on Unix systems.

Those Unix shell quoting rules differ from the quoting rules applying to
Windows' cmd and Powershell, making it a little awkward to quote
command-line parameters properly when spawning other processes.

In particular, git.exe passes arguments to subprocesses that are *not*
intended to be interpreted as wildcards, and if they contain
backslashes, those are not to be interpreted as escape characters, e.g.
when passing Windows paths.

Note: this is only a problem when calling MSYS2 executables, not when
calling MINGW executables such as git.exe. However, we do call MSYS2
executables frequently, most notably when setting the use_shell flag in
the child_process structure.

There is no elegant way to determine whether the .exe file to be
executed is an MSYS2 program or a MINGW one. But since the use case of
passing a command line through the shell is so prevalent, we need to
work around this issue at least when executing sh.exe.

Let's introduce an ugly, hard-coded test whether argv[0] is "sh", and
whether it refers to the MSYS2 Bash, to determine whether we need to
quote the arguments differently than usual.

That still does not fix the issue completely, but at least it is
something.

Incidentally, this also fixes the problem where `git clone \\server\repo`
failed due to incorrect handling of the backslashes when handing the path
to the git-upload-pack process.

Further, we need to take care to quote not only whitespace and
backslashes, but also curly brackets. As aliases frequently go through
the MSYS2 Bash, and as aliases frequently get parameters such as
HEAD@{yesterday}, this is really important. As an early version of this
patch broke this, let's make sure that this does not regress by adding a
test case for that.

Helped-by: Kim Gybels <kgybels@infogroep.be>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

mingw (t5580): document bug when cloning from backslashed UNC paths

Due to a quirk in Git's method to spawn git-upload-pack, there is a
problem when passing paths with backslashes in them: Git will force the
command-line through the shell, which has different quoting semantics in
Git for Windows (being an MSYS2 program) than regular Win32 executables
such as git.exe itself.

The symptom is that the first of the two backslashes in UNC paths of the
form \\myserver\folder\repository.git is *stripped off*.

Document this bug by introducing a test case.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ls-refs: filter refs using namespace-stripped name

If a user fetches refs/heads/master from a repo with namespace "ns", the
remote is expected to (1) not send the real refs/heads/master, and (2)
send refs/namespaces/ns/refs/heads/master with the name
refs/heads/master. (1) indeed happens now, but not (2) - Git only sends
refs that have the user-given prefix, but it checks them against the
full name of the ref (the one starting with refs/namespaces), and not
the namespace-stripped one.

This is demonstrated by the patch in the test. Currently, it results in
"fatal: couldn't find remote ref refs/heads/master" despite both
unnamespaced and namespaced master being present. With the code change,
it produces the expected result.

Check the ref prefixes against the namespace-stripped name.

This bug was discovered through applying patches [1] that override
protocol.version to 2 in repositories when running tests, allowing us to
notice differences in behavior across different protocol versions.

[1] https://public-inbox.org/git/cover.1547677183.git.jonathantanmy@google.com/

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Add 'human' date format

This adds --date=human, which skips the timezone if it matches the
current time-zone, and doesn't print the whole date if that matches (ie
skip printing year for dates that are "this year", but also skip the
whole date itself if it's in the last few days and we can just say what
weekday it was).

For really recent dates (same day), use the relative date stamp, while
for old dates (year doesn't match), don't bother with time and timezone.

Also add 'auto' date mode, which defaults to human if we're using the
pager. So you can do

git config --add log.date auto

and your "git log" commands will show the human-legible format unless
you're scripting things.

Note that this time format still shows the timezone for recent enough
events (but not so recent that they show up as relative dates). You can
combine it with the "-local" suffix to never show timezones for an even
more simplified view.

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Stephen P. Smith <ischis2@cox.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

built-in rebase: call `git am` directly

While the scripted `git rebase` still has to rely on the
`git-rebase--am.sh` script to implement the glue between the `rebase`
and the `am` commands, we can go a more direct route in the built-in
rebase and avoid using a shell script altogether.

This patch represents a straight-forward port of `git-rebase--am.sh` to
C, along with the glue code to call it directly from within
`builtin/rebase.c`.

This reduces the chances of Git for Windows running into trouble due to
problems with the POSIX emulation layer (known as "MSYS2 runtime",
itself a derivative of the Cygwin runtime): when no shell script is
called, the POSIX emulation layer is avoided altogether.

Note: we pass an empty action to `reset_head()` here when moving back to
the original branch, as no other action is applicable, really. This
parameter is used to initialize `unpack_trees()`' messages.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

rebase: teach `reset_head()` to optionally skip the worktree

This is what the legacy (scripted) rebase does in
`move_to_original_branch`, and we will need this functionality in the
next commit.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

rebase: avoid double reflog entry when switching branches

When switching a branch *and* updating said branch to a different
revision, let's avoid a double entry in HEAD's reflog by first updating
the branch and then adjusting the symbolic ref HEAD.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

rebase: move `reset_head()` into a better spot

Over the next commits, we want to make use of it in `run_am()` (i.e.
running the `--am` backend directly, without detouring to Unix shell
script code) which in turn will be called from `run_specific_rebase()`.

So let's move it before that latter function.

This commit is best viewed using --color-moved.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

abspath_part_inside_repo: respect core.ignoreCase

If the file system is case-insensitive, we really must be careful to
ignore differences in case only.

This fixes https://github.com/git-for-windows/git/issues/735

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-p4: handle update of moved/copied files when updating a shelve

Perforce requires a complete list of files being operated on. If
git is updating an existing shelved changelist, then any files
which are moved or copied were not being added to this list.

Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-p4: add failing test for shelved CL update involving move/copy

Updating a shelved P4 changelist where one or more files have
been moved or copied does not work. Add a test for this.

The problem is that P4 requires a complete list of the files being
changed, and move/copy only includes the _source_ in the case of
updating a shelved changelist. This results in errors from Perforce
such as:

//depot/src - needs tofile //depot/dst
Submit aborted -- fix problems then use 'p4 submit -c 1234'

Signed-off-by: Luke Diamand <luke@diamand.org>
Acked-by: Andrey Mazo <amazo@checkvideo.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

pack-objects: create GIT_TEST_PACK_SPARSE

Create a test variable GIT_TEST_PACK_SPARSE to enable the sparse
object walk algorithm by default during the test suite. Enabling
this variable ensures coverage in many interesting cases, such as
shallow clones, partial clones, and missing objects.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

pack-objects: create pack.useSparse setting

The '--sparse' flag in 'git pack-objects' changes the algorithm
used to enumerate objects to one that is faster for individual
users pushing new objects that change only a small cone of the
working directory. The sparse algorithm is not recommended for a
server, which likely sends new objects that appear across the
entire working directory.

Create a 'pack.useSparse' setting that enables this new algorithm.
This allows 'git push' to use this algorithm without passing a
'--sparse' flag all the way through four levels of run_command()
calls.

If the '--no-sparse' flag is set, then this config setting is
overridden.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

revision: implement sparse algorithm

When enumerating objects to place in a pack-file during 'git
pack-objects --revs', we discover the "frontier" of commits
that we care about and the boundary with commit we find
uninteresting. From that point, we walk trees to discover which
trees and blobs are uninteresting. Finally, we walk trees from the
interesting commits to find the interesting objects that are
placed in the pack.

This commit introduces a new, "sparse" way to discover the
uninteresting trees. We use the perspective of a single user trying
to push their topic to a large repository. That user likely changed
a very small fraction of the paths in their working directory, but
we spend a lot of time walking all reachable trees.

The way to switch the logic to work in this sparse way is to start
caring about which paths introduce new trees. While it is not
possible to generate a diff between the frontier boundary and all
of the interesting commits, we can simulate that behavior by
inspecting all of the root trees as a whole, then recursing down
to the set of trees at each path.

We already had taken the first step by passing an oidset to
mark_trees_uninteresting_sparse(). We now create a dictionary
whose keys are paths and values are oidsets. We consider the set
of trees that appear at each path. While we inspect a tree, we
add its subtrees to the oidsets corresponding to the tree entry's
path. We also mark trees as UNINTERESTING if the tree we are
parsing is UNINTERESTING.

To actually improve the performance, we need to terminate our
recursion. If the oidset contains only UNINTERESTING trees, then
we do not continue the recursion. This avoids walking trees that
are likely to not be reachable from interesting trees. If the
oidset contains only interesting trees, then we will walk these
trees in the final stage that collects the intersting objects to
place in the pack. Thus, we only recurse if the oidset contains
both interesting and UNINITERESTING trees.

There are a few ways that this is not a universally better option.

First, we can pack extra objects. If someone copies a subtree
from one tree to another, the first tree will appear UNINTERESTING
and we will not recurse to see that the subtree should also be
UNINTERESTING. We will walk the new tree and see the subtree as
a "new" object and add it to the pack. A test is modified to
demonstrate this behavior and to verify that the new logic is
being exercised.

Second, we can have extra memory pressure. If instead of being a
single user pushing a small topic we are a server sending new
objects from across the entire working directory, then we will
gain very little (the recursion will rarely terminate early) but
will spend extra time maintaining the path-oidset dictionaries.

Despite these potential drawbacks, the benefits of the algorithm
are clear. By adding a counter to 'add_children_by_path' and
'mark_tree_contents_uninteresting', I measured the number of
parsed trees for the two algorithms in a variety of repos.

For git.git, I used the following input:

v2.19.0
^v2.19.0~10

Objects to pack: 550
Walked (old alg): 282
Walked (new alg): 130

For the Linux repo, I used the following input:

v4.18
^v4.18~10

Objects to pack: 518
Walked (old alg): 4,836
Walked (new alg): 188

The two repos above are rather "wide and flat" compared to
other repos that I have used in the past. As a comparison,
I tested an old topic branch in the Azure DevOps repo, which
has a much deeper folder structure than the Linux repo.

Objects to pack: 220
Walked (old alg): 22,804
Walked (new alg): 129

I used the number of walked trees the main metric above because
it is consistent across multiple runs. When I ran my tests, the
performance of the pack-objects command with the same options
could change the end-to-end time by 10x depending on the file
system being warm. However, by repeating the same test on repeat
I could get more consistent timing results. The git.git and
Linux tests were too fast overall (less than 0.5s) to measure
an end-to-end difference. The Azure DevOps case was slow enough
to see the time improve from 15s to 1s in the warm case. The
cold case was 90s to 9s in my testing.

These improvements will have even larger benefits in the super-
large Windows repository. In our experiments, we see the
"Enumerate objects" phase of pack-objects taking 60-80% of the
end-to-end time of non-trivial pushes, taking longer than the
network time to send the pack and the server time to verify the
pack.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

list-objects: consume sparse tree walk

When creating a pack-file using 'git pack-objects --revs' we provide
a list of interesting and uninteresting commits. For example, a push
operation would make the local topic branch be interesting and the
known remote refs as uninteresting. We want to discover the set of
new objects to send to the server as a thin pack.

We walk these commits until we discover a frontier of commits such
that every commit walk starting at interesting commits ends in a root
commit or unintersting commit. We then need to discover which
non-commit objects are reachable from uninteresting commits. This
commit walk is not changing during this series.

The mark_edges_uninteresting() method in list-objects.c iterates on
the commit list and does the following:

* If the commit is UNINTERSTING, then mark its root tree and every
object it can reach as UNINTERESTING.

* If the commit is interesting, then mark the root tree of every
UNINTERSTING parent (and all objects that tree can reach) as
UNINTERSTING.

At the very end, we repeat the process on every commit directly
given to the revision walk from stdin. This helps ensure we properly
cover shallow commits that otherwise were not included in the
frontier.

The logic to recursively follow trees is in the
mark_tree_uninteresting() method in revision.c. The algorithm avoids
duplicate work by not recursing into trees that are already marked
UNINTERSTING.

Add a new 'sparse' option to the mark_edges_uninteresting() method
that performs this logic in a slightly different way. As we iterate
over the commits, we add all of the root trees to an oidset. Then,
call mark_trees_uninteresting_sparse() on that oidset. Note that we
include interesting trees in this process. The current implementation
of mark_trees_unintersting_sparse() will walk the same trees as
the old logic, but this will be replaced in a later change.

Add a '--sparse' flag in 'git pack-objects' to call this new logic.
Add a new test script t/t5322-pack-objects-sparse.sh that tests this
option. The tests currently demonstrate that the resulting object
list is the same as the old algorithm. This includes a case where
both algorithms pack an object that is not needed by a remote due to
limits on the explored set of trees. When the sparse algorithm is
changed in a later commit, we will add a test that demonstrates a
change of behavior in some cases.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

revision: add mark_tree_uninteresting_sparse

In preparation for a new algorithm that walks fewer trees when
creating a pack from a set of revisions, create a method that
takes an oidset of tree oids and marks reachable objects as
UNINTERESTING.

The current implementation uses the existing
mark_tree_uninteresting to recursively walk the trees and blobs.
This will walk the same number of trees as the old mechanism. To
ensure that mark_tree_uninteresting walks the tree, we need to
remove the UNINTERESTING flag before calling the method. This
implementation will be replaced entirely in a later commit.

There is one new assumption in this approach: we are also given
the oids of the interesting trees. This implementation does not
use those trees at the moment, but we will use them in a later
rewrite of this method.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

add: use separate ADD_CACHE_RENORMALIZE flag

Commit 9472935d81 (add: introduce "--renormalize", 2017-11-16) taught
git-add to pass HASH_RENORMALIZE to add_to_index(), which then passes
the flag along to index_path(). However, the flags taken by
add_to_index() and the ones taken by index_path() are distinct
namespaces. We cannot take HASH_* flags in add_to_index(), because they
overlap with the ADD_CACHE_* flags we already take (in this case,
HASH_RENORMALIZE conflicts with ADD_CACHE_IGNORE_ERRORS).

We can solve this by adding a new ADD_CACHE_RENORMALIZE flag, and using
it to set HASH_RENORMALIZE within add_to_index(). In order to make it
clear that these two flags come from distinct sets, let's also change
the name "newflags" in the function to "hash_flags".

Reported-by: Dmitriy Smirnov <dmitriy.smirnov@jetbrains.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t6042: work around speed optimization on Windows

When Git determines whether a file has changed, it looks at the mtime,
at the file size, and to detect changes even if the mtime is the same
(on Windows, the mtime granularity is 100ns, read: if two files are
written within the same 100ns time slot, they have the same mtime) and
even if the file size is the same, Git also looks at the inode/device
numbers.

This design obviously comes from a Linux background, where `lstat()`
calls were designed to be cheap.

On Windows, there is no `lstat()`. It has to be emulated. And while
obtaining the mtime and the file size is not all that expensive (you can
get both with a single `GetFileAttributesW()` call), obtaining the
equivalent of the inode and device numbers is very expensive (it
requires a call to `GetFileInformationByHandle()`, which in turn
requires a file handle, which is *a lot* more expensive than one might
imagine).

As it is very uncommon for developers to modify files within 100ns time
slots, Git for Windows chooses not to fill inode/device numbers
properly, but simply sets them to 0.

However, in t6042 the files file_v1 and file_v2 are typically written
within the same 100ns time slot, and they do not differ in file size. So
the minor modification is not picked up.

Let's work around this issue by avoiding the `git mv` calls in the
'mod6-setup: chains of rename/rename(1to2) and rename/rename(2to1)' test
case. The target files are overwritten anyway, so it is not like we
really rename those files. This fixes the issue because `git add` will
now add the files as new files (as opposed to existing, just renamed
files).

Functionally, we do not change anything because we replace two `git mv
<old> <new>` calls (where `<new>` is completely overwritten and `git
add`ed later anyway) by `git rm <old>` calls (removing other files, too,
that are also completely overwritten and `git add`ed later).

Reviewed-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tests: define GIT_TEST_SIDEBAND_ALL

Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used
from tests. When set to true, this overrides uploadpack.allowsidebandall
to true, allowing the entire test suite to be run as if this
configuration is in place for all repositories.

As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset
or set to 1.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

{fetch,upload}-pack: sideband v2 fetch response

Currently, a response to a fetch request has sideband support only while
the packfile is being sent, meaning that the server cannot send notices
until the start of the packfile.

Extend sideband support in protocol v2 fetch responses to the whole
response. upload-pack will advertise it if the
uploadpack.allowsidebandall configuration variable is set, and
fetch-pack will automatically request it if advertised.

If the sideband is to be used throughout the whole response, upload-pack
will use it to send errors instead of prefixing a PKT-LINE payload with
"ERR ".

This will be tested in a subsequent patch.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

sideband: reverse its dependency on pkt-line

A subsequent patch will teach struct packet_reader a new field that, if
set, instructs it to interpret read data as multiplexed. This will
create a dependency from pkt-line to sideband.

To avoid a circular dependency, split recv_sideband() into 2 parts: the
reading loop (left in recv_sideband()) and the processing of the
contents (in demultiplex_sideband()), and move the former into pkt-line.
This reverses the direction of dependency: sideband no longer depends on
pkt-line, and pkt-line now depends on sideband.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

travis-ci: build with the right compiler

Our 'Makefile' hardcodes the compiler to build Git as 'CC = cc'. This
CC variable can be overridden from the command line, i.e. 'make
CC=gcc-X.Y' will build with that particular GCC version, but not from
the environment, i.e. 'CC=gcc-X.Y make' will still build with whatever
'cc' happens to be on the platform.

Our build jobs on Travis CI are badly affected by this. In the build
matrix we have dedicated build jobs to build Git with GCC and Clang
both on Linux and macOS from the very beginning (522354d70f (Add
Travis CI support, 2015-11-27)). Alas, this never really worked as
supposed to, because Travis CI specifies the compiler for those build
jobs as 'export CC=gcc' and 'export CC=clang' (which works fine for
projects built with './configure && make'). Consequently, our
'linux-clang' build job has always used GCC, because that's where 'cc'
points at in Travis CI's Linux images, while the 'osx-gcc' build job
has always used Clang. Furthermore, 37fa4b3c78 (travis-ci: run gcc-8
on linux-gcc jobs, 2018-05-19) added an 'export CC=gcc-8' in an
attempt to build with a more modern compiler, but to no avail.

Set MAKEFLAGS with CC based on the $CC environment variable, so 'make'
will run the "right" compiler. The Xcode 10.1 macOS image on Travis
CI already contains the gcc@8 package from Homebrew, but we have to
'brew link' it first to be able to use it.

So with this patch our build jobs will build Git with the following
compiler versions:

linux-clang: clang version 5.0.0 (tags/RELEASE_500/final)
linux-gcc: gcc-8 (Ubuntu 8.1.0-5ubuntu1~14.04) 8.1.0

osx-clang: Apple LLVM version 10.0.0 (clang-1000.11.45.5)
osx-gcc: gcc-8 (Homebrew GCC 8.2.0) 8.2.0

GETTEXT_POISON: gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

travis-ci: switch to Xcode 10.1 macOS image

When building something with GCC installed from Homebrew in the
default macOS (with Xcode 9.4) image on Travis CI, it errors out with
something like this:

/usr/local/Cellar/gcc/8.1.0/lib/gcc/8/gcc/x86_64-apple-darwin17.5.0/8.1.0/include-fixed/stdio.h:78:10: fatal error: _stdio.h: No such file or directory
#include <_stdio.h>
^~~~~~~~~~

This seems to be a common problem affecting several projects, and the
common solution is to use a Travis CI macOS image with more recent
Xcode version, e.g. 10 or 10.1.

While we don't use such a GCC yet, in the very next patch we will, so
switch our OSX build jobs to use the Xcode 10.1 image. Compared to
the Xcode 10 image, this has the benefit that it comes with GCC (v8.2)
preinstalled from Homebrew.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

travis-ci: don't be '--quiet' when running the tests

All Travis CI build jobs run the test suite with 'make --quiet test'.

On one hand, being quiet doesn't save us from much clutter in the
output:

$ make test |wc -l
861
$ make --quiet test |wc -l
848

It only spares 13 lines, mostly the output of entering the 't/'
directory and the pre- and post-cleanup commands, which is negligible
compared to the ~700 lines printed while building Git and the ~850
lines of 'prove' output.

On the other hand, it's asking for trouble. In our CI build scripts
we build Git and run the test suite in two separate 'make'
invocations. In a prelimiary version of one of the later patches in
this series, to explicitly specify which compiler to use, I changed
them to basically run:

make CC=$CC
make --quiet test

naively thinking that it should Just Work... but then that 'make
--quiet test' got all clever on me, noticed the changed build flags,
and then proceeded to rebuild everything with the default 'cc'. And
because of that '--quiet' option, it did so, well, quietly, only
saying "* new build flags", and it was by mere luck that I happened to
notice that something is amiss.

Let's just drop that '--quiet' option when running the test suite in
all build scripts.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

.gitignore: ignore external debug symbols from GCC on macOS

When Git is build with a "real" GCC on macOS [1], or at least with GCC
installed via Homebrew, and CFLAGS includes the '-g' option (and our
default CFLAGS does), then by default GCC writes the debug symbols
into external files under '<binary>.dSYM/' directories (e.g.
'git-daemon.dSYM/', 'git.dSYM/', etc.).

Update '.gitignore' to ignore these directories, so they don't clutter
the output of 'git status'. Furthermore, these build artifacts then
won't trigger build failures on Travis CI via b92cb86ea1 (travis-ci:
check that all build artifacts are .gitignore-d, 2017-12-31) once one
of the following patches updates our CI build scripts to use a real
GCC in the 'osx-gcc' build job.

[1] On macOS the default '/usr/bin/gcc' executable is not a real GCC,
but merely a compatibility wrapper around Clang:

$ gcc --version
Configured with: --prefix=<...>
Apple LLVM version 9.0.0 (clang-900.0.39.2)
<...>

So even though 'make CC=gcc' does indeed execute a command called
'gcc', in the end Git will be built with Clang all the same.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

compat/obstack: fix -Wcast-function-type warnings

GCC 8 introduced the new -Wcast-function-type warning, which is
implied by -Wextra (which, in turn is enabled in our DEVELOPER flags).
When building Git with GCC 8 and this warning enabled on a non-glibc
platform [1], one is greeted with a screenful of compiler
warnings/errors:

compat/obstack.c: In function '_obstack_begin':
compat/obstack.c:162:17: error: cast between incompatible function types from 'void * (*)(long int)' to 'struct _obstack_chunk * (*)(void *, long int)' [-Werror=cast-function-type]
h->chunkfun = (struct _obstack_chunk * (*)(void *, long)) chunkfun;
^
compat/obstack.c:163:16: error: cast between incompatible function types from 'void (*)(void *)' to 'void (*)(void *, struct _obstack_chunk *)' [-Werror=cast-function-type]
h->freefun = (void (*) (void *, struct _obstack_chunk *)) freefun;
^
compat/obstack.c:116:8: error: cast between incompatible function types from 'struct _obstack_chunk * (*)(void *, long int)' to 'struct _obstack_chunk * (*)(long int)' [-Werror=cast-function-type]
: (*(struct _obstack_chunk *(*) (long)) (h)->chunkfun) ((size)))
^
compat/obstack.c:168:22: note: in expansion of macro 'CALL_CHUNKFUN'
chunk = h->chunk = CALL_CHUNKFUN (h, h -> chunk_size);
^~~~~~~~~~~~~
<snip>

'struct obstack' stores pointers to two functions to allocate and free
"chunks", and depending on how obstack is used, these functions take
either one parameter (like standard malloc() and free() do; this is
how we use it in 'kwset.c') or two parameters. Presumably to reduce
memory footprint, a single field is used to store the function pointer
for both signatures, and then it's casted to the appropriate signature
when the function pointer is accessed. These casts between function
pointers with different number of parameters are what trigger those
compiler errors.

Modify 'struct obstack' to use unions to store function pointers with
different signatures, and then use the union member with the
appropriate signature when accessing these function pointers. This
eliminates the need for those casts, and thus avoids this compiler
error.

[1] Compiling 'compat/obstack.c' on a platform with glibc is sort of
a noop, see the comment before '# define ELIDE_CODE', so this is
not an issue on common Linux distros.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

helper/test-ref-store: fix "new-sha1" vs "old-sha1" typo

It looks like it is a copy-paste error made in 80f2a6097c
(t/helper: add test-ref-store to test ref-store functions,
2017-03-26) to pass "old-sha1" instead of "new-sha1" to
notnull() when we get the new sha1 argument from
const char **argv.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
Acked-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Makefile: correct example fuzz build

The comment explaining how to build the fuzzers was broken in
927c77e7d4d ("Makefile: use FUZZ_CXXFLAGS for linking fuzzers",
2018-11-14).

When building fuzzers, all .c files must be compiled with coverage
tracing enabled. This is not possible when using only FUZZ_CXXFLAGS, as
that flag is only applied to the fuzzers themselves. Switching back to
CFLAGS fixes the issue.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph: fix buffer read-overflow

fuzz-commit-graph identified a case where Git will read past the end of
a buffer containing a commit graph if the graph's header has an
incorrect chunk count. A simple bounds check in parse_commit_graph()
prevents this.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph, fuzz: add fuzzer for commit-graph

Break load_commit_graph_one() into a new function, parse_commit_graph().
The latter function operates on arbitrary buffers, which makes it
suitable as a fuzzing target. Since parse_commit_graph() is only called
by load_commit_graph_one() (and the fuzzer described below), we omit
error messages that would be duplicated by the caller.

Adds fuzz-commit-graph.c, which provides a fuzzing entry point
compatible with libFuzzer (and possibly other fuzzing engines).

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

filter-options: expand scaled numbers

When communicating with a remote server or a subprocess, use
expanded numbers rather than numbers with scaling suffix in the
object filter spec (e.g. "limit:blob=1k" becomes
"limit:blob=1024").

Update the protocol docs to note that clients should always perform this
expansion, to allow for more compatibility between server
implementations.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tree:<depth>: skip some trees even when collecting omits

If a tree has already been recorded as omitted, we don't need to
traverse it again just to collect its omits. Stop traversing trees a
second time when collecting omits.

Signed-off-by: Matthew DeVore <matvore@google.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

list-objects-filter: teach tree:# how to handle >0

Implement positive values for <depth> in the tree:<depth> filter. The
exact semantics are described in Documentation/rev-list-options.txt.

The long-term goal at the end of this is to allow a partial clone to
eagerly fetch an entire directory of files by fetching a tree and
specifying <depth>=1. This, for instance, would make a build operation
fast and convenient. It is fast because the partial clone does not need
to fetch each file individually, and convenient because the user does
not need to supply a sparse-checkout specification.

Another way of considering this feature is as a way to reduce
round-trips, since the client can get any number of levels of
directories in a single request, rather than wait for each level of tree
objects to come back, whose entries are used to construct a new request.

Signed-off-by: Matthew DeVore <matvore@google.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Merge branch 'nd/the-index' into md/list-objects-filter-by-depth

pkt-line: introduce struct packet_writer

A future patch will allow the client to request multiplexing of the
entire fetch response (and not only during packfile transmission), which
in turn allows the server to send progress and keepalive messages at any
time during the response.

It will be convenient for a future patch if writing options
(specifically, whether the written data is to be multiplexed) could be
controlled from a single place, so create struct packet_writer to serve
as that place, and modify upload-pack to use it.

Currently, it only stores the output fd, but a subsequent patch will (as
described above) introduce an option to determine if the written data is
to be multiplexed.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

add --edit: truncate the patch file

If there is already a .git/ADD_EDIT.patch file, we fail to truncate it
properly, which could result in very funny errors.

Of course, this file should not be left lying around. But at least in
one case, there was a stale copy, larger than the current diff. So the
result was a corrupt diff.

Let's just truncate the file when we write it and not worry about it too
much.

Reported by J Wyman.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

cache: make oidcpy always copy GIT_MAX_RAWSZ bytes

There are some situations in which we want to store an object ID into
struct object_id without the_hash_algo necessarily being set correctly.
One such case is when cloning a repository, where we must read refs from
the remote side without having a repository from which to read the
preferred algorithm.

In this cases, we may have the_hash_algo set to SHA-1, which is the
default, but read refs into struct object_id that are SHA-256. When
copying these values, we will want to copy them completely, not just the
first 20 bytes. Consequently, make sure that oidcpy copies the maximum
number of bytes at all times, regardless of the setting of
the_hash_algo.

Since oidcpy and hashcpy are no longer functionally identical, remove
the Cocinelle object_id transformations that convert from one into the
other.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tree-walk: store object_id in a separate member

When parsing a tree, we read the object ID directly out of the tree
buffer. This is normally fine, but such an object ID cannot be used with
oidcpy, which copies GIT_MAX_RAWSZ bytes, because if we are using SHA-1,
there may not be that many bytes to copy.

Instead, store the object ID in a separate struct member. Since we can
no longer efficiently compute the path length, store that information as
well in struct name_entry. Ensure we only copy the object ID into the
new buffer if the path length is nonzero, as some callers will pass us
an empty path with no object ID following it, and we will not want to
read past the end of the buffer.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

match-trees: use hashcpy to splice trees

When we splice trees together, we operate in place on the tree buffer.
If we're using SHA-1 for the hash algorithm, we may not have a full
GIT_MAX_RAWSZ (32) bytes to copy. Consequently, it doesn't logically
make sense for us to use a struct object_id to represent this type,
since it isn't a complete object.

Represent this value as a unsigned char pointer instead and copy it when
necessary.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

match-trees: compute buffer offset correctly when splicing

Currently, the struct object_id pointer returned from tree_entry_extract
lives directly inside the parsed tree buffer. In a future commit, this
will change so that it instead points to a dedicated struct member.
Since in this code path, we want to modify the buffer directly, compute
the buffer offset we want to modify by using the pointer to the path
instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tree-walk: copy object ID before use

In a future commit, the pointer returned by tree_entry_extract will
point into the struct tree_desc, causing its lifetime to be bound to
that of the struct tree_desc itself. To ensure this code path keeps
working, copy the object_id into a local variable so that it lives long
enough.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Second batch after 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>

Merge branch 'do/gitweb-strict-export-conf-doc'

Doc update.

* do/gitweb-strict-export-conf-doc:
docs: fix $strict_export text in gitweb.conf.txt

Merge branch 'nd/indentation-fix'

Code cleanup.

* nd/indentation-fix:
Indent code with TABs

Merge branch 'en/directory-renames-nothanks-doc-update'

Doc update.

* en/directory-renames-nothanks-doc-update:
git-rebase.txt: update note about directory rename detection and am

Merge branch 'bw/mailmap'

* bw/mailmap:
mailmap: update brandon williams's email address

Merge branch 'fd/gitweb-snapshot-conf-doc-fix'

Doc update.

* fd/gitweb-snapshot-conf-doc-fix:
docs/gitweb.conf: config variable typo

Merge branch 'tb/use-common-win32-pathfuncs-on-cygwin'

Cygwin update.

* tb/use-common-win32-pathfuncs-on-cygwin:
git clone <url> C:\cygwin\home\USER\repo' is working (again)

Merge branch 'km/rebase-doc-typofix'

Doc update.

* km/rebase-doc-typofix:
rebase docs: drop stray word in merge command description

Merge branch 'md/exclude-promisor-objects-fix-cleanup'

Code clean-up.

* md/exclude-promisor-objects-fix-cleanup:
revision.c: put promisor option in specialized struct

Merge branch 'tb/log-G-binary'

"git log -G<regex>" looked for a hunk in the "git log -p" patch
output that contained a string that matches the given pattern.
Optimize this code to ignore binary files, which by default will
not show any hunk that would match any pattern (unless textconv or
the --text option is in effect, that is).

* tb/log-G-binary:
log -G: ignore binary files

Merge branch 'sb/diff-color-moved-config-option-fixup'

Minor inconsistency fix.

* sb/diff-color-moved-config-option-fixup:
diff: align move detection error handling with other options

Merge branch 'hn/highlight-sideband-keywords'

Lines that begin with a certain keyword that come over the wire, as
well as lines that consist only of one of these keywords, ought to
be painted in color for easier eyeballing, but the latter was
broken ever since the feature was introduced in 2.19, which has
been corrected.

* hn/highlight-sideband-keywords:
sideband: color lines with keyword only

Merge branch 'cb/test-lint-cp-a'

BSD port update.

* cb/test-lint-cp-a:
tests: add lint for non portable cp -a

Merge branch 'cb/t5004-empty-tar-archive-fix'

BSD port update.

* cb/t5004-empty-tar-archive-fix:
t5004: avoid using tar for empty packages

Merge branch 'cb/openbsd-allows-reading-directory'

BSD port update.

* cb/openbsd-allows-reading-directory:
config.mak.uname: OpenBSD uses BSD semantics with fread for directories

Merge branch 'hb/t0061-dot-in-path-fix'

Test update.

* hb/t0061-dot-in-path-fix:
t0061: do not fail test if '.' is part of $PATH

Merge branch 'nd/checkout-noisy'

"git checkout [<tree-ish>] path..." learned to report the number of
paths that have been checked out of the index or the tree-ish,
which gives it the same degree of noisy-ness as the case in which
the command checks out a branch.

* nd/checkout-noisy:
t0027: squelch checkout path run outside test_expect_* block
checkout: print something when checking out paths

Merge branch 'ab/commit-graph-progress-fix'

* ab/commit-graph-progress-fix:
commit-graph: split up close_reachable() progress output

Merge branch 'nd/attr-pathspec-in-tree-walk'

The traversal over tree objects has learned to honor
":(attr:label)" pathspec match, which has been implemented only for
enumerating paths on the filesystem.

* nd/attr-pathspec-in-tree-walk:
tree-walk: support :(attr) matching
dir.c: move, rename and export match_attrs()
pathspec.h: clean up "extern" in function declarations
tree-walk.c: make tree_entry_interesting() take an index
tree.c: make read_tree*() take 'struct repository *'