gitweb.git
Merge branch 'bp/rename-test-env-var'Junio C Hamano Fri, 19 Oct 2018 04:34:03 +0000 (13:34 +0900)

Merge branch 'bp/rename-test-env-var'

Some environment variables that control the runtime options of Git
used during tests are getting renamed for consistency.

* bp/rename-test-env-var:
t0000: do not get self-test disrupted by environment warnings
preload-index: update GIT_FORCE_PRELOAD_TEST support
read-cache: update TEST_GIT_INDEX_VERSION support
fsmonitor: update GIT_TEST_FSMONITOR support
preload-index: use git_env_bool() not getenv() for customization
t/README: correct spelling of "uncommon"

Merge branch 'ss/wt-status-committable'Junio C Hamano Fri, 19 Oct 2018 04:34:02 +0000 (13:34 +0900)

Merge branch 'ss/wt-status-committable'

Code clean-up in the internal machinery used by "git status" and
"git commit --dry-run".

* ss/wt-status-committable:
roll wt_status_state into wt_status and populate in the collect phase
wt-status.c: set the committable flag in the collect phase
t7501: add test of "commit --dry-run --short"
wt-status: rename commitable to committable
wt-status.c: move has_unmerged earlier in the file

Merge branch 'nd/the-index'Junio C Hamano Fri, 19 Oct 2018 04:34:02 +0000 (13:34 +0900)

Merge branch 'nd/the-index'

Various codepaths in the core-ish part learn to work on an
arbitrary in-core index structure, not necessarily the default
instance "the_index".

* nd/the-index: (23 commits)
revision.c: reduce implicit dependency the_repository
revision.c: remove implicit dependency on the_index
ws.c: remove implicit dependency on the_index
tree-diff.c: remove implicit dependency on the_index
submodule.c: remove implicit dependency on the_index
line-range.c: remove implicit dependency on the_index
userdiff.c: remove implicit dependency on the_index
rerere.c: remove implicit dependency on the_index
sha1-file.c: remove implicit dependency on the_index
patch-ids.c: remove implicit dependency on the_index
merge.c: remove implicit dependency on the_index
merge-blobs.c: remove implicit dependency on the_index
ll-merge.c: remove implicit dependency on the_index
diff-lib.c: remove implicit dependency on the_index
read-cache.c: remove implicit dependency on the_index
diff.c: remove implicit dependency on the_index
grep.c: remove implicit dependency on the_index
diff.c: remove the_index dependency in textconv() functions
blame.c: rename "repo" argument to "r"
combine-diff.c: remove implicit dependency on the_index
...

Merge branch 'nd/complete-fetch-multiple-args'Junio C Hamano Fri, 19 Oct 2018 04:34:01 +0000 (13:34 +0900)

Merge branch 'nd/complete-fetch-multiple-args'

Teach bash completion that "git fetch --multiple" only takes remote
names as arguments and no refspecs.

* nd/complete-fetch-multiple-args:
completion: support "git fetch --multiple"

commit-reach: fix first-parent heuristicDerrick Stolee Thu, 18 Oct 2018 17:24:40 +0000 (10:24 -0700)

commit-reach: fix first-parent heuristic

The algorithm in can_all_from_reach_with_flags() performs a depth-
first-search, terminated by generation number, intending to use
a hueristic that "important" commits are found in the first-parent
history. This heuristic is valuable in scenarios like fetch
negotiation.

However, there is a problem! After the search finds a target commit,
it should pop all commits off the stack and mark them as "can reach".
This logic is incorrect, so the algorithm instead walks all reachable
commits above the generation-number cutoff.

The existing algorithm is still an improvement over the previous
algorithm, as the worst-case complexity went from quadratic to linear.
The performance measurement at the time was good, but not dramatic.
By fixing this heuristic, we reduce the number of walked commits.

We can also re-run the performance tests from commit 4fbcca4e
"commit-reach: make can_all_from_reach... linear".

Performance was measured on the Linux repository using
'test-tool reach can_all_from_reach'. The input included rows seeded by
tag values. The "small" case included X-rows as v4.[0-9]* and Y-rows as
v3.[0-9]*. This mimics a (very large) fetch that says "I have all major
v3 releases and want all major v4 releases." The "large" case included
X-rows as "v4.*" and Y-rows as "v3.*". This adds all release-candidate
tags to the set, which does not greatly increase the number of objects
that are considered, but does increase the number of 'from' commits,
demonstrating the quadratic nature of the previous code.

Small Case:

4fbcca4e~1: 0.85 s
4fbcca4e: 0.26 s (num_walked: 1,011,035)
HEAD: 0.14 s (num_walked: 8,601)

Large Case:

4fbcca4e~1: 24.0 s
4fbcca4e: 0.12 s (num_walked: 503,925)
HEAD: 0.06 s (num_walked: 217,243)

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ref-filter: free item->value and item->value->sOlga Telezhnaya Thu, 18 Oct 2018 07:28:54 +0000 (07:28 +0000)

ref-filter: free item->value and item->value->s

Release item->value.
Initialize item->value->s dynamically and then release its resources.
Release some local variables.

Final goal of this patch is to reduce number of memory leaks.

Signed-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ls-remote: release memory instead of UNLEAKOlga Telezhnaya Thu, 18 Oct 2018 07:28:54 +0000 (07:28 +0000)

ls-remote: release memory instead of UNLEAK

Use ref_array_clear() to release memory instead of UNLEAK macros.

Signed-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ref-filter: free memory from used_atomOlga Telezhnaya Thu, 18 Oct 2018 07:28:54 +0000 (07:28 +0000)

ref-filter: free memory from used_atom

Release memory from used_atom variable for reducing number of memory
leaks.

Signed-off-by: Olga Telezhnaia <olyatelezhnaya@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: avoid dead store for struct progressCarlo Marcelo Arenas Belón Thu, 18 Oct 2018 18:59:20 +0000 (11:59 -0700)

multi-pack-index: avoid dead store for struct progress

it is initialized unconditionally by a call to start_progress
below.

Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

unpack-trees: avoid dead store for struct progressCarlo Marcelo Arenas Belón Thu, 18 Oct 2018 18:46:04 +0000 (11:46 -0700)

unpack-trees: avoid dead store for struct progress

it is unconditionally initialized a few lines below

Signed-off-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

config.mak.dev: enable -Wunused-functionJeff King Thu, 18 Oct 2018 07:05:22 +0000 (03:05 -0400)

config.mak.dev: enable -Wunused-function

We explicitly omitted -Wunused-function when we added
-Wextra, but there is no need: the current code compiles
cleanly with it. And it's worth having, since it can let you
know when there are cascading effects from a cleanup (e.g.,
deleting one function lets you delete its static helpers).

There are cases where we may need an unused function to
exist, but we can handle these easily:

- macro-generated code like commit-slab; there we have the
MAYBE_UNUSED annotation to silence the compiler

- conditional compilation, where we may or may not need a
static helper. These generally fall into one of two
categories:

- the call should not be conditional, but rather the
function body itself should be (and may just be a
no-op on one side of the #if). That keeps the
conditional pollution out of the main code.

- call-chains of static helpers should all be in the
same #if block, so they are all-or-nothing

And if there's some case that doesn't cover, we still
have MAYBE_UNUSED as a fallback.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ci: add optional test variablesDerrick Stolee Wed, 17 Oct 2018 13:00:34 +0000 (06:00 -0700)

ci: add optional test variables

The commit-graph and multi-pack-index features introduce optional
data structures that are not required for normal Git operations.
It is important to run the normal test suite without them enabled,
but it is helpful to also run the test suite using them.

Our continuous integration scripts include a second test stage that
runs with optional GIT_TEST_* variables enabled. Add the following
two variables to that stage:

GIT_TEST_COMMIT_GRAPH
GIT_TEST_MULTI_PACK_INDEX

This will slow down the operation, as we build a commit-graph file
after every 'git commit' operation and build a multi-pack-index
during every 'git repack' operation. However, it is important that
future changes are compatible with these features.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

headers: normalize the spelling of some header guardsRamsay Jones Wed, 17 Oct 2018 22:13:26 +0000 (23:13 +0100)

headers: normalize the spelling of some header guards

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

list-objects: support for skipping tree traversalMatthew DeVore Thu, 18 Oct 2018 00:39:15 +0000 (17:39 -0700)

list-objects: support for skipping tree traversal

The tree:0 filter does not need to traverse the trees that it has
filtered out, so optimize list-objects and list-objects-filter to skip
traversing the trees entirely. Before this patch, we iterated over all
children of the tree, and did nothing for all of them, which was
wasteful.

Signed-off-by: Matthew DeVore <matvore@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

test-tool: show tool list on errorJeff King Wed, 17 Oct 2018 09:25:07 +0000 (05:25 -0400)

test-tool: show tool list on error

Before we switched to one big test-tool binary, if you
forgot the name of a tool, you could use tab-completion in
the shell to get a hint. But these days, all you get is:

$ t/helper/test-tool approxidate
fatal: There is no test named 'approxidate'

and you're stuck reading the source code to find it. Let's
print a list of the available tools in this case.

Signed-off-by: Jeff King <peff@peff.net>
Reviewed-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: fix small typo in git show-branchSaulius Gurklys Wed, 17 Oct 2018 05:37:41 +0000 (08:37 +0300)

doc: fix small typo in git show-branch

Fix small typo as in document <glob> is used not <globs>.

Signed-off-by: Saulius Gurklys <s4uliu5@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

builtin/submodule--helper: remove debugging leftover... Stefan Beller Tue, 16 Oct 2018 23:45:50 +0000 (16:45 -0700)

builtin/submodule--helper: remove debugging leftover tracing

I noticed 74d4731da1 (submodule--helper: replace connect-gitdir-workingtree
by ensure-core-worktree, 2018-08-13) had two leftover debugging statements
when reading The coverage report [1]. Remove them.

https://public-inbox.org/git/e30a9c05-87d8-1f2b-182c-6d6a5fefe43c@gmail.com/

Signed-off-by: Stefan Beller <sbeller@google.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

branch: trivial style fixTao Qingyun Tue, 16 Oct 2018 14:19:20 +0000 (22:19 +0800)

branch: trivial style fix

Signed-off-by: Tao Qingyun <taoqy@ls-a.me>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Revert "subtree: make install targets depend on build... Junio C Hamano Thu, 18 Oct 2018 02:07:17 +0000 (11:07 +0900)

Revert "subtree: make install targets depend on build targets"

This reverts commit 744f7c4c314dc0e7816ac05520e8358c8318187a.

These targets do depend on the fact that each prereq is explicitly
listed via their use of $^, which I failed to notice, and broke the
build.

builtin/branch.c: remove useless branch_getTao Qingyun Tue, 16 Oct 2018 14:54:28 +0000 (22:54 +0800)

builtin/branch.c: remove useless branch_get

branch_get sometimes returns current_branch, which can be NULL (e.g., if
you're on a detached HEAD). Try:

$ git branch HEAD
fatal: no such branch 'HEAD'

$ git branch ''
fatal: no such branch ''

However, it seems weird that we'd check those cases here (and provide
such lousy messages). And indeed, dropping that and letting us
eventually hit create_branch() gives a much better message:

$ git branch HEAD
fatal: 'HEAD' is not a valid branch name.

$ git branch ''
fatal: '' is not a valid branch name.

Signed-off-by: Tao Qingyun <taoqy@ls-a.me>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

subtree: make install targets depend on build targetsChristian Hesse Tue, 16 Oct 2018 07:56:24 +0000 (09:56 +0200)

subtree: make install targets depend on build targets

Now that we have build targets let the install targets depend on them.
Also make the targets phony.

Signed-off-by: Christian Hesse <mail@eworm.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

send-email: also pick up cc addresses from -by trailersRasmus Villemoes Tue, 16 Oct 2018 07:39:23 +0000 (09:39 +0200)

send-email: also pick up cc addresses from -by trailers

When rerolling a patch series, including various Reviewed-by etc. that
may have come in, it is quite convenient to have git-send-email
automatically cc those people.

So pick up any *-by lines, with a new suppression category 'misc-by',
but special-case Signed-off-by, since that already has its own
suppression category. It seems natural to make 'misc-by' implied by
'body'.

Based-on-patch-by: Joe Perches <joe@perches.com>
Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Fourth batch for 2.20Junio C Hamano Tue, 16 Oct 2018 07:21:17 +0000 (16:21 +0900)

Fourth batch for 2.20

Signed-off-by: Junio C Hamano <gitster@pobox.com>

Merge branch 'sf/complete-stash-list'Junio C Hamano Tue, 16 Oct 2018 07:16:09 +0000 (16:16 +0900)

Merge branch 'sf/complete-stash-list'

The completion script (in contrib/) learned to complete a handful of
options "git stash list" command takes.

* sf/complete-stash-list:
git-completion.bash: add completion for stash list

Merge branch 'mw/doc-typofixes'Junio C Hamano Tue, 16 Oct 2018 07:16:09 +0000 (16:16 +0900)

Merge branch 'mw/doc-typofixes'

Typofixes.

* mw/doc-typofixes:
docs: typo: s/isimilar/similar/
docs: graph: remove unnecessary `graph_update()' call
docs: typo: s/go/to/

Merge branch 'js/mingw-wants-vista-or-above'Junio C Hamano Tue, 16 Oct 2018 07:16:08 +0000 (16:16 +0900)

Merge branch 'js/mingw-wants-vista-or-above'

The minimum version of Windows supported by Windows port fo Git is
now set to Vista.

* js/mingw-wants-vista-or-above:
mingw: bump the minimum Windows version to Vista
mingw: set _WIN32_WINNT explicitly for Git for Windows
compat/poll: prepare for targeting Windows Vista

Merge branch 'rs/sequencer-oidset-insert-avoids-dups'Junio C Hamano Tue, 16 Oct 2018 07:16:08 +0000 (16:16 +0900)

Merge branch 'rs/sequencer-oidset-insert-avoids-dups'

Code clean-up.

* rs/sequencer-oidset-insert-avoids-dups:
sequencer: use return value of oidset_insert()

Merge branch 'jk/oideq-hasheq-cleanup'Junio C Hamano Tue, 16 Oct 2018 07:16:07 +0000 (16:16 +0900)

Merge branch 'jk/oideq-hasheq-cleanup'

Code clean-up.

* jk/oideq-hasheq-cleanup:
more oideq/hasheq conversions

Merge branch 'ma/mailing-list-address-in-git-help'Junio C Hamano Tue, 16 Oct 2018 07:16:07 +0000 (16:16 +0900)

Merge branch 'ma/mailing-list-address-in-git-help'

Doc update.

* ma/mailing-list-address-in-git-help:
git doc: direct bug reporters to mailing list archive

Merge branch 'nd/packobjectshook-doc-fix'Junio C Hamano Tue, 16 Oct 2018 07:16:07 +0000 (16:16 +0900)

Merge branch 'nd/packobjectshook-doc-fix'

Doc update.

* nd/packobjectshook-doc-fix:
config.txt: correct the note about uploadpack.packObjectsHook

Merge branch 'rt/rebase-typofix'Junio C Hamano Tue, 16 Oct 2018 07:16:06 +0000 (16:16 +0900)

Merge branch 'rt/rebase-typofix'

Typofix.

* rt/rebase-typofix:
git-rebase.sh: fix typos in error messages

Merge branch 'ma/t1400-undebug-test'Junio C Hamano Tue, 16 Oct 2018 07:16:06 +0000 (16:16 +0900)

Merge branch 'ma/t1400-undebug-test'

Test fix.

* ma/t1400-undebug-test:
t1400: drop debug `echo` to actually execute `test`

Merge branch 'ma/commit-graph-docs'Junio C Hamano Tue, 16 Oct 2018 07:16:05 +0000 (16:16 +0900)

Merge branch 'ma/commit-graph-docs'

Doc update.

* ma/commit-graph-docs:
Doc: refer to the "commit-graph file" with dash
git-commit-graph.txt: refer to "*commit*-graph file"
git-commit-graph.txt: typeset more in monospace
git-commit-graph.txt: fix bullet lists

Merge branch 'dz/credential-doc-url-matching-rules'Junio C Hamano Tue, 16 Oct 2018 07:16:05 +0000 (16:16 +0900)

Merge branch 'dz/credential-doc-url-matching-rules'

Doc update.

* dz/credential-doc-url-matching-rules:
doc: clarify gitcredentials path component matching

Merge branch 'en/status-multiple-renames-to-the-same... Junio C Hamano Tue, 16 Oct 2018 07:16:05 +0000 (16:16 +0900)

Merge branch 'en/status-multiple-renames-to-the-same-target-fix'

The code in "git status" sometimes hit an assertion failure. This
was caused by a structure that was reused without cleaning the data
used for the first run, which has been corrected.

* en/status-multiple-renames-to-the-same-target-fix:
commit: fix erroneous BUG, 'multiple renames on the same target? how?'

Merge branch 'ds/reachable-final-cleanup'Junio C Hamano Tue, 16 Oct 2018 07:16:04 +0000 (16:16 +0900)

Merge branch 'ds/reachable-final-cleanup'

Code already in 'master' is further cleaned-up by this patch.

* ds/reachable-final-cleanup:
commit-reach: cleanups in can_all_from_reach...

Merge branch 'jk/check-everything-connected-is-long... Junio C Hamano Tue, 16 Oct 2018 07:16:04 +0000 (16:16 +0900)

Merge branch 'jk/check-everything-connected-is-long-gone'

Comment fix.

* jk/check-everything-connected-is-long-gone:
receive-pack: update comment with check_everything_connected

Merge branch 'jn/gc-auto'Junio C Hamano Tue, 16 Oct 2018 07:16:02 +0000 (16:16 +0900)

Merge branch 'jn/gc-auto'

"gc --auto" ended up calling exit(-1) upon error, which has been
corrected to use exit(1). Also the error reporting behaviour when
daemonized has been updated to exit with zero status when stopping
due to a previously discovered error (which implies there is no
point running gc to improve the situation); we used to exit with
failure in such a case.

* jn/gc-auto:
gc: do not return error for prior errors in daemonized mode

Merge branch 'jn/gc-auto-prep'Junio C Hamano Tue, 16 Oct 2018 07:16:02 +0000 (16:16 +0900)

Merge branch 'jn/gc-auto-prep'

Code clean-up.

* jn/gc-auto-prep:
gc: exit with status 128 on failure
gc: improve handling of errors reading gc.log

Merge branch 'md/test-cleanup'Junio C Hamano Tue, 16 Oct 2018 07:16:01 +0000 (16:16 +0900)

Merge branch 'md/test-cleanup'

Various test scripts have been updated for style and also correct
handling of exit status of various commands.

* md/test-cleanup:
tests: order arguments to git-rev-list properly
t9109: don't swallow Git errors upstream of pipes
tests: don't swallow Git errors upstream of pipes
t/*: fix ordering of expected/observed arguments
tests: standardize pipe placement
Documentation: add shell guidelines
t/README: reformat Do, Don't, Keep in mind lists

Merge branch 'fe/doc-updates'Junio C Hamano Tue, 16 Oct 2018 07:16:01 +0000 (16:16 +0900)

Merge branch 'fe/doc-updates'

Doc updates.

* fe/doc-updates:
git-describe.1: clarify that "human readable" is also git-readable
git-column.1: clarify initial description, provide examples
git-archimport.1: specify what kind of Arch we're talking about

Merge branch 'jn/mailmap-update'Junio C Hamano Tue, 16 Oct 2018 07:16:01 +0000 (16:16 +0900)

Merge branch 'jn/mailmap-update'

The mailmap file update.

* jn/mailmap-update:
mailmap: consistently normalize brian m. carlson's name

Merge branch 'tg/t5551-with-curl-7.61.1'Junio C Hamano Tue, 16 Oct 2018 07:16:00 +0000 (16:16 +0900)

Merge branch 'tg/t5551-with-curl-7.61.1'

Test update.

* tg/t5551-with-curl-7.61.1:
t5551: compare sorted cookies files
t5551: move setup code inside test_expect blocks

Merge branch 'en/merge-cleanup'Junio C Hamano Tue, 16 Oct 2018 07:16:00 +0000 (16:16 +0900)

Merge branch 'en/merge-cleanup'

Code clean-up.

* en/merge-cleanup:
merge-recursive: rename merge_file_1() and merge_content()
merge-recursive: remove final remaining caller of merge_file_one()
merge-recursive: avoid wrapper function when unnecessary and wasteful
merge-recursive: set paths correctly when three-way merging content

Merge branch 'rj/header-check'Junio C Hamano Tue, 16 Oct 2018 07:16:00 +0000 (16:16 +0900)

Merge branch 'rj/header-check'

Header files clean-up.

* rj/header-check:
delta-islands.h: add missing forward declarations (hdr-check)
midx.h: add missing forward declarations (hdr-check)
refs/refs-internal.h: add missing declarations (hdr-check)
refs/packed-backend.h: add missing declaration (hdr-check)
refs/ref-cache.h: add missing declarations (hdr-check)
ewah/ewok_rlw.h: add missing include (hdr-check)
json-writer.h: add missing include (hdr-check)
Makefile: add a hdr-check target

Merge branch 'ma/config-doc-update'Junio C Hamano Tue, 16 Oct 2018 07:16:00 +0000 (16:16 +0900)

Merge branch 'ma/config-doc-update'

Doc update.

* ma/config-doc-update:
git-config.txt: fix 'see: above' note
Doc: use `--type=bool` instead of `--bool`

Merge branch 'jk/delta-islands-with-bitmap-reuse-delta... Junio C Hamano Tue, 16 Oct 2018 07:16:00 +0000 (16:16 +0900)

Merge branch 'jk/delta-islands-with-bitmap-reuse-delta-fix'

Fix interactions between two recent topics.

* jk/delta-islands-with-bitmap-reuse-delta-fix:
pack-objects: handle island check for "external" delta base

Merge branch 'tq/refs-internal-comment-fix'Junio C Hamano Tue, 16 Oct 2018 07:15:59 +0000 (16:15 +0900)

Merge branch 'tq/refs-internal-comment-fix'

Fix for typo in a sample code in comment.

* tq/refs-internal-comment-fix:
refs: docstring typo

Merge branch 'ts/alias-of-alias'Junio C Hamano Tue, 16 Oct 2018 07:15:59 +0000 (16:15 +0900)

Merge branch 'ts/alias-of-alias'

An alias that expands to another alias has so far been forbidden,
but now it is allowed to create such an alias.

* ts/alias-of-alias:
t0014: introduce an alias testing suite
alias: show the call history when an alias is looping
alias: add support for aliases of an alias

Merge branch 'ds/commit-graph-with-grafts'Junio C Hamano Tue, 16 Oct 2018 07:15:59 +0000 (16:15 +0900)

Merge branch 'ds/commit-graph-with-grafts'

The recently introduced commit-graph auxiliary data is incompatible
with mechanisms such as replace & grafts that "breaks" immutable
nature of the object reference relationship. Disable optimizations
based on its use (and updating existing commit-graph) when these
incompatible features are in use in the repository.

* ds/commit-graph-with-grafts:
commit-graph: close_commit_graph before shallow walk
commit-graph: not compatible with uninitialized repo
commit-graph: not compatible with grafts
commit-graph: not compatible with replace objects
test-repository: properly init repo
commit-graph: update design document
refs.c: upgrade for_each_replace_ref to be a each_repo_ref_fn callback
refs.c: migrate internal ref iteration to pass thru repository argument

Merge branch 'ab/commit-graph-progress'Junio C Hamano Tue, 16 Oct 2018 07:15:58 +0000 (16:15 +0900)

Merge branch 'ab/commit-graph-progress'

Generation of (experimental) commit-graph files have so far been
fairly silent, even though it takes noticeable amount of time in a
meaningfully large repository. The users will now see progress
output.

* ab/commit-graph-progress:
gc: fix regression in 7b0f229222 impacting --quiet
commit-graph verify: add progress output
commit-graph write: add progress output

git-p4: fully support unshelving changelistsLuke Diamand Mon, 15 Oct 2018 11:14:08 +0000 (12:14 +0100)

git-p4: fully support unshelving changelists

The previous git-p4 unshelve support would check for changes
in Perforce to the files being unshelved since the original
shelve, and would complain if any were found.

This was to ensure that the user wouldn't end up with both the
shelved change delta, and some deltas from other changes in their
git commit.

e.g. given fileA:
the
quick
brown
fox

change1: s/the/The/ <- p4 shelve this change
change2: s/fox/Fox/ <- p4 submit this change
git p4 unshelve 1 <- FAIL

This change teaches the P4Unshelve class to always create a parent
commit which matches the P4 tree (for the files being unshelved) at
the point prior to the P4 shelve being created (which is reported
in the p4 description for a shelved changelist).

That then means git-p4 can always create a git commit matching the
P4 shelve that was originally created, without any extra deltas.

The user might still need to use the --origin option though - there
is no way for git-p4 to work out the versions of all of the other
*unchanged* files in the shelve, since this information is not recorded
by Perforce.

Additionally this fixes handling of shelved 'move' operations.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-p4: unshelve into refs/remotes/p4-unshelved, not... Luke Diamand Mon, 15 Oct 2018 11:14:07 +0000 (12:14 +0100)

git-p4: unshelve into refs/remotes/p4-unshelved, not refs/remotes/p4/unshelved

The branch detection code looks for branches under refs/remotes/p4/...
and can end up getting confused if there are unshelved changes in
there as well. This happens in the function p4BranchesInGit().

Instead, put the unshelved changes into refs/remotes/p4-unshelved/<N>.

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

mingw: use domain information for default emailJohannes Schindelin Mon, 15 Oct 2018 09:47:08 +0000 (02:47 -0700)

mingw: use domain information for default email

When a user is registered in a Windows domain, it is really easy to
obtain the email address. So let's do that.

Suggested by Lutz Roeder.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

getpwuid(mingw): provide a better default for the user... Johannes Schindelin Mon, 15 Oct 2018 09:47:06 +0000 (02:47 -0700)

getpwuid(mingw): provide a better default for the user name

We do have the excellent GetUserInfoEx() function to obtain more
detailed information of the current user (if the user is part of a
Windows domain); Let's use it.

Suggested by Lutz Roeder.

To avoid the cost of loading Secur32.dll (even lazily, loading DLLs
takes a non-neglibile amount of time), we use the established technique
to load DLLs only when, and if, needed.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

getpwuid(mingw): initialize the structure only onceJohannes Schindelin Mon, 15 Oct 2018 09:47:05 +0000 (02:47 -0700)

getpwuid(mingw): initialize the structure only once

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fuzz: add fuzz testing for packfile indices.Josh Steadmon Sat, 13 Oct 2018 00:58:41 +0000 (17:58 -0700)

fuzz: add fuzz testing for packfile indices.

Breaks the majority of check_packed_git_idx() into a separate function,
load_idx(). The latter function operates on arbitrary buffers, which
makes it suitable as a fuzzing test target.

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fuzz: add basic fuzz testing target.Josh Steadmon Sat, 13 Oct 2018 00:58:40 +0000 (17:58 -0700)

fuzz: add basic fuzz testing target.

fuzz-pack-headers.c provides a fuzzing entry point compatible with
libFuzzer (and possibly other fuzzing engines).

Signed-off-by: Josh Steadmon <steadmon@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

rerere: convert to use the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:02:02 +0000 (00:02 +0000)

rerere: convert to use the_hash_algo

Since this data is stored in the .git directory, it makes sense for us
to use the same hash algorithm for it as for everything else. Convert
the remaining uses of SHA-1 to use the_hash_algo. Use GIT_MAX_RAWSZ for
allocations. Rename various struct members, local variables, and a
function to be named "hash" instead of "sha1".

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

submodule: make zero-oid comparison hash function agnosticbrian m. carlson Mon, 15 Oct 2018 00:02:01 +0000 (00:02 +0000)

submodule: make zero-oid comparison hash function agnostic

With SHA-256, the length of the all-zeros object ID is longer. Add a
function to git-submodule.sh to check if a full hex object ID is the
all-zeros value, and use it to check the output we're parsing from git
diff-files or diff-index.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

apply: rename new_sha1_prefix and old_sha1_prefixbrian m. carlson Mon, 15 Oct 2018 00:02:00 +0000 (00:02 +0000)

apply: rename new_sha1_prefix and old_sha1_prefix

Rename these structure members to "new_oid_prefix" and "old_oid_prefix".

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

apply: replace hard-coded constantsbrian m. carlson Mon, 15 Oct 2018 00:01:59 +0000 (00:01 +0000)

apply: replace hard-coded constants

Replace several 40-based constants with references to GIT_MAX_HEXSZ or
the_hash_algo, as appropriate.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

tag: express constant in terms of the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:01:58 +0000 (00:01 +0000)

tag: express constant in terms of the_hash_algo

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

transport: use parse_oid_hex instead of a constantbrian m. carlson Mon, 15 Oct 2018 00:01:57 +0000 (00:01 +0000)

transport: use parse_oid_hex instead of a constant

Use parse_oid_hex to compute a pointer instead of using GIT_SHA1_HEXSZ.
This simplifies the code and makes it independent of the hash length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

upload-pack: express constants in terms of the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:01:56 +0000 (00:01 +0000)

upload-pack: express constants in terms of the_hash_algo

Convert all uses of the GIT_SHA1_HEXSZ to use the_hash_algo so that they
are appropriate for any given hash length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

refs/packed-backend: express constants using the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:01:55 +0000 (00:01 +0000)

refs/packed-backend: express constants using the_hash_algo

Switch uses of GIT_SHA1_HEXSZ to use the_hash_algo so that they are
appropriate for the any given hash length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

packfile: express constants in terms of the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:01:54 +0000 (00:01 +0000)

packfile: express constants in terms of the_hash_algo

Replace uses of GIT_SHA1_RAWSZ with references to the_hash_algo to avoid
dependence on a particular hash length.

It's likely that in the future, we'll update the pack format to indicate
what hash algorithm it uses, and then this code will change. However,
at least on an interim basis, make it easier to develop on a pure
SHA-256 Git by using the_hash_algo here.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

pack-revindex: express constants in terms of the_hash_algobrian m. carlson Mon, 15 Oct 2018 00:01:53 +0000 (00:01 +0000)

pack-revindex: express constants in terms of the_hash_algo

Express the various constants used in terms of the_hash_algo.
While we're at it, fix a comment style issue as well.

Reviewed-by: Stefan Beller <sbeller@google.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

builtin/fetch-pack: remove constants with parse_oid_hexbrian m. carlson Mon, 15 Oct 2018 00:01:52 +0000 (00:01 +0000)

builtin/fetch-pack: remove constants with parse_oid_hex

Instead of using GIT_SHA1_HEXSZ, use parse_oid_hex to compute a pointer
and use that in comparisons. This is both simpler to read and works
independent of the hash length. Update references to SHA-1 in the same
function to refer to object IDs instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

builtin/mktree: remove hard-coded constantbrian m. carlson Mon, 15 Oct 2018 00:01:51 +0000 (00:01 +0000)

builtin/mktree: remove hard-coded constant

Instead of using a hard-coded constant for the size of a hex object ID,
switch to use the computed pointer from parse_oid_hex that points after
the parsed object ID.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

builtin/repack: replace hard-coded constantsbrian m. carlson Mon, 15 Oct 2018 00:01:50 +0000 (00:01 +0000)

builtin/repack: replace hard-coded constants

Note that while the error messages here are not translated, the end user
should never see them. We invoke git pack-objects shortly before both
invocations, so we can be fairly certain that the data we're receiving
is in fact valid.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

pack-bitmap-write: use GIT_MAX_RAWSZ for allocationbrian m. carlson Mon, 15 Oct 2018 00:01:49 +0000 (00:01 +0000)

pack-bitmap-write: use GIT_MAX_RAWSZ for allocation

Reviewed-by: Stefan Beller <sbeller@google.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

object_id.cocci: match only expressions of type 'struct... SZEDER Gábor Mon, 15 Oct 2018 00:01:48 +0000 (00:01 +0000)

object_id.cocci: match only expressions of type 'struct object_id'

Most of our semantic patches in 'contrib/coccinelle/object_id.cocci'
turn calls of SHA1-specific functions into calls of their
corresponding object_id counterparts, e.g. sha1_to_hex() to
oid_to_hex(). These semantic patches look something like this:

@@
expression E1;
@@
- sha1_to_hex(E1.hash)
+ oid_to_hex(&E1)

and match the access to the 'hash' field in any data type, not only in
'struct object_id', and, consquently, can produce wrong
transformations.

Case in point is the recent hash function transition patch "rerere:
convert to use the_hash_algo" [1], which, among other things, renamed
'struct rerere_dir's 'sha1' field to 'hash', and then 'make
coccicheck' started to suggest the following wrong transformations for
'rerere.c' [2]:

- return sha1_to_hex(id->collection->hash);
+ return oid_to_hex(id->collection);

and

- DIR *dir = opendir(git_path("rr-cache/%s", sha1_to_hex(rr_dir->hash)));
+ DIR *dir = opendir(git_path("rr-cache/%s", oid_to_hex(rr_dir)));

Avoid such wrong transformations by tightening semantic patches in
'object_id.cocci' to match only type of or pointers to 'struct
object_id'.

[1] https://public-inbox.org/git/20181008215701.779099-15-sandals@crustytoothpaste.net/
[2] https://travis-ci.org/git/git/jobs/440463476#L580

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

filter-trees: code clean-up of testsMatthew DeVore Fri, 12 Oct 2018 20:01:41 +0000 (13:01 -0700)

filter-trees: code clean-up of tests

A few trivial updates to test to match the current best practices.

- avoid "grep -q" that strips potentially useful output from tests
running under "-v".

- use test_write_lines to prepare multi-line expected output file.

- reserve use of test_must_fail to "git" commands.

Signed-off-by: Matthew DeVore <matvore@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

subtree: performance improvement for finding unexpected... Roger Strain Fri, 12 Oct 2018 13:52:18 +0000 (08:52 -0500)

subtree: performance improvement for finding unexpected parent commits

After testing a previous patch at larger scale, a performance issue was
detected when using git show to locate parent revisions, with a single
run of the git show command taking 2 seconds or longer in a complex repo.
When the command is required tens or hundreds of times in a run of the
script, the additional wait time is unaccepatable. Replacing the command
with git rev-parse resulted in significantly increased performance, with
the command in question returning instantly.

Signed-off-by: Roger Strain <rstrain@swri.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff.c: pass sign_index to emit_line_ws_markupStefan Beller Wed, 10 Oct 2018 23:24:59 +0000 (16:24 -0700)

diff.c: pass sign_index to emit_line_ws_markup

Instead of passing the sign directly to emit_line_ws_markup, pass only the
index to lookup the sign in diff_options->output_indicators.

Signed-off-by: Stefan Beller <sbeller@google.com>
Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-p4: do not fail in verbose mode for missing 'fileSi... Luke Diamand Fri, 12 Oct 2018 05:28:31 +0000 (06:28 +0100)

git-p4: do not fail in verbose mode for missing 'fileSize' key

If deleting or moving a file, sometimes P4 doesn't report the file size.

The code handles this just fine but some logging crashes. Stop this
happening.

There was some earlier discussion on the list about this:

https://public-inbox.org/git/xmqq1sqpp1vv.fsf@gitster.mtv.corp.google.com/

Signed-off-by: Luke Diamand <luke@diamand.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

log: fix coloring of certain octopus merge shapesNoam Postavsky Sun, 2 Sep 2018 00:07:16 +0000 (20:07 -0400)

log: fix coloring of certain octopus merge shapes

For octopus merges where the first parent edge immediately merges into
the next column to the left, the number of columns should be one less
than the usual case.

First parent to the left case:

| *-.
| |\ \
|/ / /

The usual case:

| *-.
| |\ \
| | | *

Also refactor the code to iterate over columns rather than dashes,
building from an initial patch suggested by Jeff King.

Signed-off-by: Noam Postavsky <npostavs@users.sourceforge.net>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: move git-cherry to plumbingDaniels Umanovskis Thu, 11 Oct 2018 18:33:50 +0000 (20:33 +0200)

doc: move git-cherry to plumbing

Also remove git-cherry from Bash completion because plumbing
commands do not belong there.

Signed-off-by: Daniels Umanovskis <daniels@umanovskis.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: move git-get-tar-commit-id to plumbingDaniels Umanovskis Thu, 11 Oct 2018 18:39:32 +0000 (20:39 +0200)

doc: move git-get-tar-commit-id to plumbing

This is definitely a low-level command, it's hard to argue
against it belonging in plumbing.

Signed-off-by: Daniels Umanovskis <daniels@umanovskis.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

split-index: BUG() when cache entry refers to non-exist... SZEDER Gábor Thu, 11 Oct 2018 09:53:57 +0000 (11:53 +0200)

split-index: BUG() when cache entry refers to non-existing shared entry

When the split index feature is in use, then a cache entry is:

- either only present in the split index, in which case its 'index'
field must be 0,

- or it should refer to an existing entry in the shared index, i.e.
the 'index' field can't be greater than the size of the shared
index.

If a cache entry were to refer to a non-existing entry in the shared
index, then that's a sign of something being wrong in the index state,
either as a result of a bug in dealing with the split/shared index
entries, or perhaps a (potentially unrelated) memory corruption issue.

prepare_to_write_split_index() already has a condition to catch cache
entries with such bogus 'index' field, but instead of calling BUG() it
just sets cache entry's 'index = 0', and the entry will then be
written to the new split index.

Don't write a new index file from bogus index state, and call BUG()
upon encountering an cache entry referring to a non-existing shared
index entry.

Running the test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes'
doesn't trigger this condition.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

split-index: smudge and add racily clean cache entries... SZEDER Gábor Thu, 11 Oct 2018 09:43:09 +0000 (11:43 +0200)

split-index: smudge and add racily clean cache entries to split index

Ever since the split index feature was introduced [1], refreshing a
split index is prone to a variant of the classic racy git problem.

Consider the following sequence of commands updating the split index
when the shared index contains a racily clean cache entry, i.e. an
entry whose cached stat data matches with the corresponding file in
the worktree and the cached mtime matches that of the index:

echo "cached content" >file
git update-index --split-index --add file
echo "dirty worktree" >file # size stays the same!
# ... wait ...
git update-index --add other-file

Normally, when a non-split index is updated, then do_write_index()
(the function responsible for writing all kinds of indexes, "regular",
split, and shared) recognizes racily clean cache entries, and writes
them with smudged stat data, i.e. with file size set to 0. When
subsequent git commands read the index, they will notice that the
smudged stat data doesn't match with the file in the worktree, and
then go on to check the file's content and notice its dirtiness.

In the above example, however, in the second 'git update-index'
prepare_to_write_split_index() decides which cache entries stored only
in the shared index should be replaced in the new split index. Alas,
this function never looks out for racily clean cache entries, and
since the file's stat data in the worktree hasn't changed since the
shared index was written, it won't be replaced in the new split index.
Consequently, do_write_index() doesn't even get this racily clean
cache entry, and can't smudge its stat data. Subsequent git commands
will then see that the index has more recent mtime than the file and
that the (not smudged) cached stat data still matches with the file in
the worktree, and, ultimately, will erroneously consider the file
clean.

Modify prepare_to_write_split_index() to recognize racily clean cache
entries, and mark them to be added to the split index. Note that
there are two places where it should check raciness: first those cache
entries that are only stored in the shared index, and then those that
have been copied by unpack_trees() from the shared index while it
constructed a new index. This way do_write_index() will get these
racily clean cache entries as well, and will then write them with
smudged stat data to the new split index.

This change makes all tests in 't1701-racy-split-index.sh' pass, so
flip the two 'test_expect_failure' tests to success. Also add the '#'
(as in nr. of trial) to those tests' description that were omitted
when the tests expected failure.

Note that after this change if the index is split when it contains a
racily clean cache entry, then a smudged cache entry will be written
both to the new shared and to the new split indexes. This doesn't
affect regular git commands: as far as they are concerned this is just
an entry in the split index replacing an outdated entry in the shared
index. It did affect a few tests in 't1700-split-index.sh', though,
because they actually check which entries are stored in the split
index; a previous patch in this series has already made the necessary
adjustments in 't1700'. And racily clean cache entries and index
splitting are rare enough to not worry about the resulting duplicated
smudged cache entries, and the additional complexity required to
prevent them is not worth it.

Several tests failed occasionally when the test suite was run with
'GIT_TEST_SPLIT_INDEX=yes'. Here are those that I managed to trace
back to this racy split index problem, starting with those failing
more frequently, with a link to a failing Travis CI build job for
each. The highlighted line [2] shows when the racy file was written,
which is not always in the failing test but in a preceeding setup
test.

t3903-stash.sh:
https://travis-ci.org/git/git/jobs/385542084#L5858

t4024-diff-optimize-common.sh:
https://travis-ci.org/git/git/jobs/386531969#L3174

t4015-diff-whitespace.sh:
https://travis-ci.org/git/git/jobs/360797600#L8215

t2200-add-update.sh:
https://travis-ci.org/git/git/jobs/382543426#L3051

t0090-cache-tree.sh:
https://travis-ci.org/git/git/jobs/416583010#L3679

There might be others, e.g. perhaps 't1000-read-tree-m-3way.sh' and
others using 'lib-read-tree-m-3way.sh', but I couldn't confirm yet.

[1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge
branch 'nd/split-index', 2014-07-16).

[2] Note that those highlighted lines are in the 'after failure' fold,
and your browser might unhelpfully fold it up before you could
take a good look.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

split-index: don't compare cached data of entries alrea... SZEDER Gábor Thu, 11 Oct 2018 09:43:08 +0000 (11:43 +0200)

split-index: don't compare cached data of entries already marked for split index

When unpack_trees() constructs a new index, it copies cache entries
from the original index [1]. prepare_to_write_split_index() has to
deal with this, and it has a dedicated code path for copied entries
that are present in the shared index, where it compares the cached
data in the corresponding copied and original entries. If the cached
data matches, then they are considered the same; if it differs, then
the copied entry will be marked for inclusion as a replacement entry
in the just about to be written split index by setting the
CE_UPDATE_IN_BASE flag.

However, a cache entry already has its CE_UPDATE_IN_BASE flag set upon
reading the split index, if the entry already has a replacement entry
there, or upon refreshing the cached stat data, if the corresponding
file was modified. The state of this flag is then preserved when
unpack_trees() copies a cache entry from the shared index.

So modify prepare_to_write_split_index() to check the copied cache
entries' CE_UPDATE_IN_BASE flag first, and skip the thorough
comparison of cached data if the flag is already set. Those couple of
lines comparing the cached data would then have too many levels of
indentation, so extract them into a helper function.

Note that comparing the cached data in copied and original entries in
the shared index might actually be entirely unnecessary. In theory
all code paths refreshing the cached stat data of an entry in the
shared index should set the CE_UPDATE_IN_BASE flag in that entry, and
unpack_trees() should preserve this flag when copying cache entries.
This means that the cached data is only ever changed if the
CE_UPDATE_IN_BASE flag is set as well. Our test suite seems to
confirm this: instrumenting the conditions in question and running the
test suite repeatedly with 'GIT_TEST_SPLIT_INDEX=yes' showed that the
cached data in a copied entry differs from the data in the shared
entry only if its CE_UPDATE_IN_BASE flag is indeed set.

In practice, however, our test suite doesn't have 100% coverage,
GIT_TEST_SPLIT_INDEX is inherently random, and I certainly can't claim
to possess complete understanding of what goes on in unpack_trees()...
Therefore I kept the comparison of the cached data when
CE_UPDATE_IN_BASE is not set, just in case that an unnoticed or future
code path were to accidentally miss setting this flag upon refreshing
the cached stat data or unpack_trees() were to drop this flag while
copying a cache entry.

[1] Note that when unpack_trees() constructs the new index and decides
that a cache entry should now refer to different content than what
was recorded in the original index (e.g. 'git read-tree -m
HEAD^'), then that can't really be considered a copy of the
original, but rather the creation of a new entry. Notably and
pertinent to the split index feature, such a new entry doesn't
have a reference to the original's shared index entry anymore,
i.e. its 'index' field is set to 0. Consequently, such an entry
is treated by prepare_to_write_split_index() as an entry not
present in the shared index and it will be added to the new split
index, while the original entry will be marked as deleted, and
neither the above discussion nor the changes in this patch apply
to them.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

split-index: count the number of deleted entriesSZEDER Gábor Thu, 11 Oct 2018 09:43:07 +0000 (11:43 +0200)

split-index: count the number of deleted entries

'struct split_index' contains the field 'nr_deletions', whose name
with the 'nr_' prefix suggests that it contains the number of deleted
cache entries. However, barring its initialization to 0, this field
is only ever set to 1, indicating that there is at least one deleted
entry, but not the number of deleted entries. Luckily, this doesn't
cause any issues (other than confusing the reader, that is), because
the only place reading this field uses it in the same sense, i.e.: 'if
(si->nr_deletions)'.

To avoid confusion, we could either rename this field to something
like 'has_deletions' to make its name match its role, or make it a
counter of deleted cache entries to match its name.

Let's make it a counter, to keep it in sync with the related field
'nr_replacements', which does contain the number of replaced cache
entries. This will also give developers debugging the split index
code easy access to the number of deleted cache entries.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1700-split-index: date back files to avoid racy situationsSZEDER Gábor Thu, 11 Oct 2018 09:43:06 +0000 (11:43 +0200)

t1700-split-index: date back files to avoid racy situations

't1700-split-index.sh' checks that the index was split correctly under
various circumstances and that all the different ways to turn the
split index feature on and off work correctly. To do so, most of its
tests use 'test-tool dump-split-index' to see which files have their
cache entries in the split index. All these tests assume that all
cache entries are written to the shared index (called "base"
throughout these tests) when a new shared index is created. This is
an implementation detail: most git commands (basically all except 'git
update-index') don't care or know at all about split index or whether
a cache entry is stored in the split or shared index.

As demonstrated in the previous patch, refreshing a split index is
prone to a variant of the classic racy git issue. The next patch will
fix this issue, but while doing so it will also slightly change this
behaviour: only cache entries with mtime in the past will be written
only to the newly created shared index, but racily clean cache entries
will be written to the new split index (with smudged stat data).

While this upcoming change won't at all affect any git commands, it
will violate the above mentioned assumption of 't1700's tests. Since
these tests create or modify files and create or refresh the split
index in rapid succession, there are plenty of racily clean cache
entries to be dealt with, which will then be written to the new split
indexes, and, ultimately, will cause several tests in 't1700' to fail.

Let's prepare 't1700-split-index.sh' for this upcoming change and
modify its tests to avoid racily clean files by backdating the mtime
of any file modifications (and since a lot of tests create or modify
files, encapsulate it into a helper function).

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

split-index: add tests to demonstrate the racy split... SZEDER Gábor Thu, 11 Oct 2018 09:43:05 +0000 (11:43 +0200)

split-index: add tests to demonstrate the racy split index problem

Ever since the split index feature was introduced [1], refreshing a
split index is prone to a variant of the classic racy git problem.
There are a couple of unrelated tests in the test suite that
occasionally fail when run with 'GIT_TEST_SPLIT_INDEX=yes', but
't1700-split-index.sh', the only test script focusing solely on split
index, has never noticed this issue, because it only cares about how
the index is split under various circumstances and all the different
ways to turn the split index feature on and off.

Add a dedicated test script 't1701-racy-split-index.sh' to exercise
the split index feature in racy situations as well; kind of a
"t0010-racy-git.sh for split index" but with modern style (the tests
do everything in &&-chained list of commands in 'test_expect_...'
blocks, and use 'test_cmp' for more informative output on failure).

The tests cover the following sequences of index splitting, updating,
and racy file modifications, with the last two cases demonstrating the
racy split index problem:

1. Split the index while adding a racily clean file:

echo "cached content" >file
git update-index --split-index --add file
echo "dirty worktree" >file # size stays the same

This case already works properly. Even though the cache entry's
stat data matches with the modifid file in the worktree,
subsequent git commands will notice that the (split) index and
the file have the same mtime, and then will go on to check the
file's content and notice its dirtiness.

2. Add a racily clean file to an already split index:

git update-index --split-index
echo "cached content" >file
git update-index --add file
echo "dirty worktree" >file

This case already works properly. After the second 'git
update-index' writes the newly added file's cache entry to the
new split index, it basically works in the same way as case #1.

3. Split the index when it (i.e. the not yet splitted index)
contains a racily clean cache entry, i.e. an entry whose cached
stat data matches with the corresponding file in the worktree and
the cached mtime matches that of the index:

echo "cached content" >file
git update-index --add file
echo "dirty worktree" >file
# ... wait ...
git update-index --split-index --add other-file

This case already works properly. The shared index is written by
do_write_index(), i.e. the same function that is responsible for
writing "regular" and split indexes as well. This function
cleverly notices the racily clean cache entry, and writes the
entry to the new shared index with smudged stat data, i.e. file
size set to 0. When subsequent git commands read the index, they
will notice that the smudged stat data doesn't match with the
file in the worktree, and then go on to check the file's content
and notice its dirtiness.

4. Update the split index when it contains a racily clean cache
entry:

git update-index --split-index
echo "cached content" >file
git update-index --add file
echo "dirty worktree" >file
# ... wait ...
git update-index --add other-file

This case already works properly. After the second 'git
update-index' the newly added file's cache entry is only stored
in the split index. If a cache entry is present in the split
index (even if it is a replacement of an outdated entry in the
shared index), then it will always be included in the new split
index on subsequent split index updates (until the file is
removed or a new shared index is written), independently from
whether the entry is racily clean or not. When do_write_index()
writes the new split index, it notices the racily clean cache
entry, and smudges its stat date. Subsequent git commands
reading the index will notice the smudged stat data and then go
on to check the file's content and notice its dirtiness.

5. Update the split index when a racily clean cache entry is stored
only in the shared index:

echo "cached content" >file
git update-index --split-index --add file
echo "dirty worktree" >file
# ... wait ...
git update-index --add other-file

This case fails due to the racy split index problem. In the
second 'git update-index' prepare_to_write_split_index() decides,
among other things, which cache entries stored only in the shared
index should be replaced in the new split index. Alas, this
function never looks out for racily clean cache entries, and
since the file's stat data in the worktree hasn't changed since
the shared index was written, the entry won't be replaced in the
new split index. Consequently, do_write_index() doesn't even get
this racily clean cache entry, and can't smudge its stat data.
Subsequent git commands will then see that the index has more
recent mtime than the file and that the (not smudged) cached stat
data still matches with the file in the worktree, and,
ultimately, will erroneously consider the file clean.

6. Update the split index after unpack_trees() copied a racily clean
cache entry from the shared index:

echo "cached content" >file
git update-index --split-index --add file
echo "dirty worktree" >file
# ... wait ...
git read-tree -m HEAD

This case fails due to the racy split index problem. This
basically fails for the same reason as case #5 above, but there
is one important difference, which warrants the dedicated test.
While that second 'git update-index' in case #5 updates
index_state in place, in this case 'git read-tree -m' calls
unpack_trees(), which throws out the entire index, and constructs
a new one from the (potentially updated) copies of the original's
cache entries. Consequently, when prepare_to_write_split_index()
gets to work on this reconstructed index, it takes a different
code path than in case #5 when deciding which cache entries in
the shared index should be replaced. The result is the same,
though: the racily clean cache entry goes unnoticed, it isn't
added to the split index with smudged stat data, and subsequent
git commands will then erroneously consider the file clean.

Note that in the last two 'test_expect_failure' cases I omitted the
'#' (as in nr. of trial) from the tests' description on purpose for
now, as it breakes the TAP output [2]; it will be added at the end of
the series, when those two tests will be flipped to
'test_expect_success'.

[1] In the branch leading to the merge commit v2.1.0-rc0~45 (Merge
branch 'nd/split-index', 2014-07-16).

[2] In the TAP output a '#' should separate the test's description
from the TODO directive emitted by 'test_expect_failure'. The
additional '#' in "#$trial" interferes with this, the test harness
won't recognize the TODO directive, and will report that those
tests failed unexpectedly.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: move git-rev-parse from porcelain to plumbingDaniels Umanovskis Wed, 10 Oct 2018 21:37:26 +0000 (23:37 +0200)

doc: move git-rev-parse from porcelain to plumbing

git-rev-parse mostly seems like plumbing, and is more usd in
scripts than in regular use. Online it's often mentioned as
a plumbing command. Nonetheless it's listed under porcelain
interrogators in `man git`. It seems appropriate to formally
move git-rev-parse to plumbing interrogators.

Signed-off-by: Daniels Umanovskis <daniels@umanovskis.se>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

gc doc: mention the commit-graph in the introÆvar Arnfjörð Bjarmason Wed, 10 Oct 2018 19:38:18 +0000 (19:38 +0000)

gc doc: mention the commit-graph in the intro

Explicitly mention in the intro that we may be writing supplemental
data structures such as the commit-graph during "gc", i.e. to call out
the "optimize" part of what this command does, it doesn't just
"collect garbage" as the "gc" name might imply.

Past changes have updated the intro to reflect new commands, such as
mentioning "worktree" in b586a96a39 ("gc.txt: more details about what
gc does", 2018-03-15). So let's elaborate on what was added in
d5d5d7b641 ("gc: automatically write commit-graph files", 2018-06-27).

See also
https://public-inbox.org/git/87tvm3go42.fsf@evledraar.gmail.com/ (follow-up
replies) for an on-list discussion about what "gc" does.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

read-cache: load cache entries on worker threadsBen Peart Wed, 10 Oct 2018 15:59:38 +0000 (11:59 -0400)

read-cache: load cache entries on worker threads

This patch helps address the CPU cost of loading the index by utilizing
the Index Entry Offset Table (IEOT) to divide loading and conversion of
the cache entries across multiple threads in parallel.

I used p0002-read-cache.sh to generate some performance data:

Test w/100,000 files reduced the time by 32.24%
Test w/1,000,000 files reduced the time by -4.77%

Note that on the 1,000,000 files case, multi-threading the cache entry parsing
does not yield a performance win. This is because the cost to parse the
index extensions in this repo, far outweigh the cost of loading the cache
entries.

The high cost of parsing the index extensions is driven by the cache tree
and the untracked cache extensions. As this is currently the longest pole,
any reduction in this time will reduce the overall index load times so is
worth further investigation in another patch series.

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ieot: add Index Entry Offset Table (IEOT) extensionBen Peart Wed, 10 Oct 2018 15:59:37 +0000 (11:59 -0400)

ieot: add Index Entry Offset Table (IEOT) extension

This patch enables addressing the CPU cost of loading the index by adding
additional data to the index that will allow us to efficiently multi-
thread the loading and conversion of cache entries.

It accomplishes this by adding an (optional) index extension that is a
table of offsets to blocks of cache entries in the index file. To make
this work for V4 indexes, when writing the cache entries, it periodically
"resets" the prefix-compression by encoding the current entry as if the
path name for the previous entry is completely different and saves the
offset of that entry in the IEOT. Basically, with V4 indexes, it
generates offsets into blocks of prefix-compressed entries.

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

read-cache: load cache extensions on a worker threadBen Peart Wed, 10 Oct 2018 15:59:36 +0000 (11:59 -0400)

read-cache: load cache extensions on a worker thread

This patch helps address the CPU cost of loading the index by loading
the cache extensions on a worker thread in parallel with loading the cache
entries.

In some cases, loading the extensions takes longer than loading the
cache entries so this patch utilizes the new EOIE to start the thread to
load the extensions before loading all the cache entries in parallel.

This is possible because the current extensions don't access the cache
entries in the index_state structure so are OK that they don't all exist
yet.

The CACHE_EXT_TREE, CACHE_EXT_RESOLVE_UNDO, and CACHE_EXT_UNTRACKED
extensions don't even get a pointer to the index so don't have access to the
cache entries.

CACHE_EXT_LINK only uses the index_state to initialize the split index.
CACHE_EXT_FSMONITOR only uses the index_state to save the fsmonitor last
update and dirty flags.

I used p0002-read-cache.sh to generate some performance data:

Test w/100,000 files reduced the time by 0.53%
Test w/1,000,000 files reduced the time by 27.78%

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

config: add new index.threads config settingBen Peart Wed, 10 Oct 2018 15:59:35 +0000 (11:59 -0400)

config: add new index.threads config setting

Add support for a new index.threads config setting which will be used to
control the threading code in do_read_index(). A value of 0 will tell the
index code to automatically determine the correct number of threads to use.
A value of 1 will make the code single threaded. A value greater than 1
will set the maximum number of threads to use.

For testing purposes, this setting can be overwritten by setting the
GIT_TEST_INDEX_THREADS=<n> environment variable to a value greater than 0.

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

eoie: add End of Index Entry (EOIE) extensionBen Peart Wed, 10 Oct 2018 15:59:34 +0000 (11:59 -0400)

eoie: add End of Index Entry (EOIE) extension

The End of Index Entry (EOIE) is used to locate the end of the variable
length index entries and the beginning of the extensions. Code can take
advantage of this to quickly locate the index extensions without having
to parse through all of the index entries.

The EOIE extension is always written out to the index file including to
the shared index when using the split index feature. Because it is always
written out, the SHA checksums in t/t1700-split-index.sh were updated
to reflect its inclusion.

It is written as an optional extension to ensure compatibility with other
git implementations that do not yet support it. It is always written out
to ensure it is available as often as possible to speed up index operations.

Because it must be able to be loaded before the variable length cache
entries and other index extensions, this extension must be written last.
The signature for this extension is { 'E', 'O', 'I', 'E' }.

The extension consists of:

- 32-bit offset to the end of the index entries

- 160-bit SHA-1 over the extension types and their sizes (but not
their contents). E.g. if we have "TREE" extension that is N-bytes
long, "REUC" extension that is M-bytes long, followed by "EOIE",
then the hash would be:

SHA-1("TREE" + <binary representation of N> +
"REUC" + <binary representation of M>)

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

read-cache: clean up casting and byte decodingBen Peart Wed, 10 Oct 2018 15:59:33 +0000 (11:59 -0400)

read-cache: clean up casting and byte decoding

This patch does a clean up pass to minimize the casting required to work
with the memory mapped index (mmap).

It also makes the decoding of network byte order more consistent by using
get_be32() where possible.

Signed-off-by: Ben Peart <benpeart@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

send-email: only consider lines containing @ or <>... Rasmus Villemoes Wed, 10 Oct 2018 11:13:50 +0000 (13:13 +0200)

send-email: only consider lines containing @ or <> for automatic Cc'ing

While the address sanitizations routines do accept local addresses, that
is almost never what is meant in a Cc or Signed-off-by trailer.

Looking through all the signed-off-by lines in the linux kernel tree
without a @, there are mostly two patterns: Either just a full name, or
a full name followed by <user at domain.com> (i.e., with the word at
instead of a @), and minor variations. For cc lines, the same patterns
appear, along with lots of "cc stable" variations that do not actually
name stable@vger.kernel.org

Cc: stable # introduced pre-git times
cc: stable.kernel.org

In the <user at domain.com> cases, one gets a chance to interactively
fix it. But when there is no <> pair, it seems we end up just using the
first word as a (local) address.

As the number of cases where a local address really was meant is
likely (and anecdotally) quite small compared to the number of cases
where we end up cc'ing a garbage address, insist on at least a @ or a <>
pair being present.

This is also preparation for the next patch, where we are likely to
encounter even more non-addresses in -by lines, such as

Reported-by: Coverity
Patch-generated-by: Coccinelle
Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Documentation/git-send-email.txt: style fixesRasmus Villemoes Wed, 10 Oct 2018 11:13:49 +0000 (13:13 +0200)

Documentation/git-send-email.txt: style fixes

For consistency, add full stops in a few places and outdent a line by
one space.

Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-help.txt: document "git help cmd" vs "git cmd ... Rasmus Villemoes Tue, 9 Oct 2018 11:59:09 +0000 (13:59 +0200)

git-help.txt: document "git help cmd" vs "git cmd --help" for aliases

This documents the existing behaviour of "git help cmd" when cmd is an
alias, as well as providing a hint to use the "git cmd --help" form to
be taken directly to the man page for the aliased command.

Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git.c: handle_alias: prepend alias info when first... Rasmus Villemoes Tue, 9 Oct 2018 11:59:08 +0000 (13:59 +0200)

git.c: handle_alias: prepend alias info when first argument is -h

Most git commands respond to -h anywhere in the command line, or at
least as a first and lone argument, by printing the usage
information. For aliases, we can provide a little more information that
might be useful in interpreting/understanding the following output by
prepending a line telling that the command is an alias, and for what.

When one invokes a simple alias, such as "cp = cherry-pick"
with -h, this results in

$ git cp -h
'cp' is aliased to 'cherry-pick'
usage: git cherry-pick [<options>] <commit-ish>...
...

When the alias consists of more than one word, this provides the
additional benefit of informing the user which options are implicit in
using the alias, e.g. with "cp = cherry-pick -n":

$ git cp -h
'cp' is aliased to 'cherry-pick -n'
usage: git cherry-pick [<options>] <commit-ish>...
...

For shell commands, we cannot know how it responds to -h, but printing
this line to stderr should not hurt, and can help in figuring out what
is happening in a case like

$ git sc -h
'sc' is aliased to '!somecommand'
somecommand: invalid option '-h'

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

help: redirect to aliased commands for "git cmd --help"Rasmus Villemoes Tue, 9 Oct 2018 11:59:07 +0000 (13:59 +0200)

help: redirect to aliased commands for "git cmd --help"

As discussed in the thread for v1 of this patch [1] [2], this changes the
rules for "git foo --help" when foo is an alias.

(1) When invoked as "git help foo", we continue to print the "foo is
aliased to bar" message and nothing else.

(2) If foo is an alias for a shell command, print "foo is aliased to
!bar" as usual.

(3) Otherwise, print "foo is aliased to bar" to the standard error
stream, and then break the alias string into words and pretend as if
"git word[0] --help" were called.

Getting the man page for git-cherry-pick directly with "git cp --help"
is consistent with "--help" generally providing more comprehensive help
than "-h". Printing the alias definition to stderr means that in certain
cases (e.g. if help.format=web or if the pager uses an alternate screen
and does not clear the terminal), one has

'cp' is aliased to 'cherry-pick -n'

above the prompt when one returns to the terminal/quits the pager, which
is a useful reminder that using 'cp' has some flag implicitly set. There
are cases where this information vanishes or gets scrolled
away, but being printed to stderr, it should never hurt.

[1] https://public-inbox.org/git/20180926102636.30691-1-rv@rasmusvillemoes.dk/
[2] https://public-inbox.org/git/20180926184914.GC30680@sigill.intra.peff.net/

Signed-off-by: Rasmus Villemoes <rv@rasmusvillemoes.dk>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

doc: fix a typo and clarify a sentenceMihir Mehta Wed, 10 Oct 2018 22:26:54 +0000 (17:26 -0500)

doc: fix a typo and clarify a sentence

I noticed that git-merge-base was unlikely to actually be a git command,
and tried it in my shell. Seeing that it doesn't work, I cleaned up two
places in the docs where it appears.

Signed-off-by: Mihir Mehta <mihir@cs.utexas.edu>
Signed-off-by: Junio C Hamano <gitster@pobox.com>