add: do not accept pathspec magic 'attr'

Commit b0db704652 (pathspec: allow querying for attributes -
2017-03-13) adds new pathspec magic 'attr' but only with
match_pathspec(). "git add" has some pathspec related code that still
does not know about 'attr' and will bail out:

$ git add ':(attr:foo)'
fatal: BUG:dir.c:1584: unsupported magic 40

A better solution would be making this code support 'attr'. But I
don't know how much work is needed (I'm not familiar with this new
magic). For now, let's simply reject this magic with a friendlier
message:

$ git add ':(attr:foo)'
fatal: :(attr:foo): pathspec magic not supported by this command: 'attr'

Update t6135 so that the expected error message is from the
"graceful" rejection codepath, not "oops, we were supposed to reject
the request to trigger this magic" codepath.

Reported-by: smaudet@sebastianaudet.com
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Merge branch 'ab/fetch-tags-noclobber'

The rules used by "git push" and "git fetch" to determine if a ref
can or cannot be updated were inconsistent; specifically, fetching
to update existing tags were allowed even though tags are supposed
to be unmoving anchoring points. "git fetch" was taught to forbid
updates to existing tags without the "--force" option.
This is a backward incompatible change but in a good way; it may
still need to be treated carefully.

* ab/fetch-tags-noclobber:
fetch doc: correct grammar in --force docs
push doc: add spacing between two words

Merge branch 'bp/checkout-new-branch-optim'

"git checkout -b newbranch [HEAD]" should not have to do as much as
checking out a commit different from HEAD. An attempt is made to
optimize this special case.

* bp/checkout-new-branch-optim:
config doc: add missing list separator for checkout.optimizeNewBranch

merge-recursive: rename merge_file_1() and merge_content()

Summary:
merge_file_1() -> merge_mode_and_contents()
merge_content() -> handle_content_merge()

merge_file_1() is a very unhelpful name. Rename it to
merge_mode_and_contents() to reflect what it does.

merge_content() calls merge_mode_and_contents() to do the main part of
its work, but most of this function was about higher level stuff, e.g.
printing out conflict messages, updating skip_worktree bits, checking
for ability to avoid updating the working tree or for D/F conflicts
being in the way, etc. Since there are several handle_*() functions for
similar levels of checking and handling in merge-recursive.c (e.g.
handle_change_delete(), handle_rename_rename_2to1()), let's rename this
function to handle_content_merge().

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

merge-recursive: remove final remaining caller of merge_file_one()

The function names merge_file_one() and merge_file_1() aren't
particularly intuitive function names, especially since there is no
associated merge_file() function that these are related to. The
previous commit showed that merge_file_one() was prone to be called when
merge_file_1() should be, and since it is just a thin wrapper around
merge_file_1() anyway and only has one caller left, let's just remove
merge_file_one() entirely.

(It also turns out that the one remaining caller of merge_file_one()
has very broken code that needs to be completely rewritten, but that's
the subject of a future patch series; for now, we're just translating
it into a merge_file_1() call.)

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

merge-recursive: avoid wrapper function when unnecessary and wasteful

merge_file_one() is a convenience function taking a bunch of oids and
modes, combining each pair into a diff_filespec, and then calling
merge_file_1(). When we already start with diff_filespec's, we can
just call merge_file_1() directly instead of splitting out the oids
and modes for the wrapper to recombine into what we already had.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

merge-recursive: set paths correctly when three-way merging content

merge_3way() has code to mark different sides of the conflict with info
about where the content comes from. If the names of the files involved
match, it simply uses the branch name. If the names of the files do not
match, it uses branchname:filename. Unfortunately, merge_content()
previously always called it with one.path = a.path = b.path. Granted,
it didn't have other path information available to it for years, but
that was corrected by passing rename_conflict_info in commit
3c217c077a86 ("merge-recursive: Provide more info in conflict markers
with file renames", 2011-08-11). In that commit, instead of just fixing
the bug with the pathnames, it created fake branch names incorporating
both the branch name and file name.

This "fake branch" workaround was extended further when I pulled that
logic out into a special function in commit dac4741554e7
("merge-recursive: Create function for merging with branchname:file
markers", 2011-08-11), and a number of other sites outside of
merge_content() have been added which call into that. However, this
Rube-Goldberg-esque setup is not merely duplicate code and unnecessary
work, it also risked having other callsites invoke it in a way that
would result in markers of the form branchname:filename:filename (i.e.
with the filename repeated).

Fix this whole mess by:
- setting one.path, a.path, and b.path appropriately
- calling merge_file_1() directly
- deleting the merge_file_special_markers() workaround wrapper

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

gc: fix regression in 7b0f229222 impacting --quiet

Fix a regression in my recent 7b0f229222 ("commit-graph write: add
progress output", 2018-09-17). The newly added progress output for
"commit-graph write" didn't check the --quiet option.

Do so, and add a test asserting that this works as expected. Since the
TTY prequisite isn't available everywhere let's add a version of this
that both requires and doesn't require that. This test might be overly
specific and will break if new progress output is added, but I think
it'll serve as a good reminder to test the undertested progress
mode(s).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Helped-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

git-config.txt: fix 'see: above' note

Rather than saying "(see: above)", drop the colon. Also drop the comma
before this note.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Doc: use `--type=bool` instead of `--bool`

After fb0dc3bac1 (builtin/config.c: support `--type=<type>` as preferred
alias for `--<type>`, 2018-04-18) we have a more modern way of spelling
`--bool`.

Update all instances except those that explicitly document the
"historical options" in git-config.txt. The other old-style
type-specifiers already seem to be gone except for in that list of
historical options.

Tweak the grammar a little in config.txt while we are there.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

delta-islands.h: add missing forward declarations (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

midx.h: add missing forward declarations (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Reviewed-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

refs/refs-internal.h: add missing declarations (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

refs/packed-backend.h: add missing declaration (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

refs/ref-cache.h: add missing declarations (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

ewah/ewok_rlw.h: add missing include (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

json-writer.h: add missing include (hdr-check)

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Makefile: add a hdr-check target

Commit ef3ca95475 ("Add missing includes and forward declarations",
2018-08-15) resulted from the author employing a manual method to
create a C file consisting of a pair of pre-processor #include
lines (for 'git-compat-util.h' and a given toplevel header), and
fixing any resulting compiler errors or warnings.

Add a Makefile target to automate this process. This implementation
relies on the '-include' and '-xc' arguments to the 'gcc' and 'clang'
compilers, which allows us to effectively create the required C
compilation unit on-the-fly. This limits the portability of this
solution to those systems which have such a compiler.

The new 'hdr-check' target can be used to check most header files in
the project (for various reasons, the 'compat' and 'xdiff' directories
are not included). Also, note that individual header files can be
checked directly using the '.hco' extension (read: Hdr-Check Object)
like so:

$ make config.hco
HDR config.h
$

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fetch doc: correct grammar in --force docs

Correct a grammar error (saying "the receiving" made no sense) in the
recently landed documentation added in my 0bc8d71b99 ("fetch: stop
clobbering existing tags without --force", 2018-08-31) by rephrasing
the sentence. Also correct 'fetching work the same way' by s/work/&s/;

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

push doc: add spacing between two words

Fix a formatting error introduced in my recently landed
fe802bd21e ("push doc: correct lies about how push refspecs work",
2018-08-31).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

config doc: add missing list separator for checkout.optimizeNewBranch

The documentation added in fa655d8411 ("checkout: optimize "git
checkout -b <new_branch>"", 2018-08-16) didn't add the double-colon
needed for the labeled list separator, as a result the added
documentation all got squashed into one paragraph. Fix that by adding
the list separator.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

pack-objects: handle island check for "external" delta base

Two recent topics, jk/pack-delta-reuse-with-bitmap and
cc/delta-islands, can have a funny interaction. When
checking if we can reuse an on-disk delta, the first topic
allows base_entry to be NULL when we find an object that's
not in the packing list. But the latter topic introduces a
call to in_same_island(), which needs to look at
base_entry->idx.oid. When these two features are used
together, we might try to dereference a NULL base_entry.

In practice, this doesn't really happen. We'd generally only
use delta islands when packing to disk, since the whole
point is to optimize the pack for serving fetches later. And
the new delta-reuse code relies on having used reachability
bitmaps to determine the set of objects, which we would
typically only do when serving an actual fetch.

However, it is technically possible to combine these
features. And even without doing so, building with
"SANITIZE=address,undefined" will cause t5310.46 to
complain. Even though that test does not have delta islands
enabled, we still take the address of the NULL entry to pass
to in_same_island(). That function then promptly returns
without dereferencing the value when it sees that islands
are not enabled, but it's enough to trigger a sanitizer
error.

The solution is straight-forward: when both features are
used together, we should pass the oid of the found base to
in_same_island().

This is tricky to do inside a single "if" statement. And
after the merge in f3504ea3dd (Merge branch
'cc/delta-islands', 2018-09-17), that "if" condition is
already getting pretty unwieldy. So this patch moves the
logic into a helper function, where we can easily use
multiple return paths. The result is a bit longer, but the
logic should be much easier to follow.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Initial batch post 2.19

Merge branch 'nd/bisect-show-list-fix'

Debugging aid update.

* nd/bisect-show-list-fix:
bisect.c: make show_list() build again

Merge branch 'ab/fetch-tags-noclobber'

The rules used by "git push" and "git fetch" to determine if a ref
can or cannot be updated were inconsistent; specifically, fetching
to update existing tags were allowed even though tags are supposed
to be unmoving anchoring points. "git fetch" was taught to forbid
updates to existing tags without the "--force" option.

* ab/fetch-tags-noclobber:
fetch: stop clobbering existing tags without --force
fetch: document local ref updates with/without --force
push doc: correct lies about how push refspecs work
push doc: move mention of "tag <tag>" later in the prose
push doc: remove confusing mention of remote merger
fetch tests: add a test for clobbering tag behavior
push tests: use spaces in interpolated string
push tests: make use of unused $1 in test description
fetch: change "branch" to "reference" in --force -h output

Merge branch 'es/worktree-forced-ops-fix'

Fix a bug in which the same path could be registered under multiple
worktree entries if the path was missing (for instance, was removed
manually). Also, as a convenience, expand the number of cases in
which --force is applicable.

* es/worktree-forced-ops-fix:
doc-diff: force worktree add
worktree: delete .git/worktrees if empty after 'remove'
worktree: teach 'remove' to override lock when --force given twice
worktree: teach 'move' to override lock when --force given twice
worktree: teach 'add' to respect --force for registered but missing path
worktree: disallow adding same path multiple times
worktree: prepare for more checks of whether path can become worktree
worktree: generalize delete_git_dir() to reduce code duplication
worktree: move delete_git_dir() earlier in file for upcoming new callers
worktree: don't die() in library function find_worktree()

Merge branch 'sg/doc-trace-appends'

Docfix.

* sg/doc-trace-appends:
Documentation/git.txt: clarify that GIT_TRACE=/path appends

Merge branch 'jk/diff-rendered-docs'

Dev doc update.

* jk/diff-rendered-docs:
Revert "doc/Makefile: drop doc-diff worktree and temporary files on "make clean""
doc/Makefile: drop doc-diff worktree and temporary files on "make clean"
doc-diff: add --clean mode to remove temporary working gunk
doc-diff: fix non-portable 'man' invocation
doc-diff: always use oids inside worktree
SubmittingPatches: mention doc-diff

Merge branch 'jk/patch-corrupted-delta-fix'

Malformed or crafted data in packstream can make our code attempt
to read or write past the allocated buffer and abort, instead of
reporting an error, which has been fixed.

* jk/patch-corrupted-delta-fix:
t5303: use printf to generate delta bases
patch-delta: handle truncated copy parameters
patch-delta: consistently report corruption
patch-delta: fix oob read
t5303: test some corrupt deltas
test-delta: read input into a heap buffer

Merge branch 'ds/commit-graph-tests'

We can now optionally run tests with commit-graph enabled.

* ds/commit-graph-tests:
commit-graph: define GIT_TEST_COMMIT_GRAPH

Merge branch 'jk/pack-objects-with-bitmap-fix'

Hotfix of the base topic.

* jk/pack-objects-with-bitmap-fix:
pack-bitmap: drop "loaded" flag
traverse_bitmap_commit_list(): don't free result
t5310: test delta reuse with bitmaps
bitmap_has_sha1_in_uninteresting(): drop BUG check

Merge branch 'rs/mailinfo-format-flowed'

"git mailinfo" used in "git am" learned to make a best-effort
recovery of a patch corrupted by MUA that sends text/plain with
format=flawed option.

* rs/mailinfo-format-flowed:
mailinfo: support format=flowed

Merge branch 'jk/cocci'

spatch transformation to replace boolean uses of !hashcmp() to
newly introduced oideq() is added, and applied, to regain
performance lost due to support of multiple hash algorithms.

* jk/cocci:
show_dirstat: simplify same-content check
read-cache: use oideq() in ce_compare functions
convert hashmap comparison functions to oideq()
convert "hashcmp() != 0" to "!hasheq()"
convert "oidcmp() != 0" to "!oideq()"
convert "hashcmp() == 0" to hasheq()
convert "oidcmp() == 0" to oideq()
introduce hasheq() and oideq()
coccinelle: use <...> for function exclusion

Merge branch 'tg/rerere-doc-updates'

Clarify a part of technical documentation for rerere.

* tg/rerere-doc-updates:
rerere: add note about files with existing conflict markers
rerere: mention caveat about unmatched conflict markers

Merge branch 'es/format-patch-rangediff'

"git format-patch" learned a new "--range-diff" option to explain
the difference between this version and the previous attempt in
the cover letter (or after the tree-dashes as a comment).

* es/format-patch-rangediff:
format-patch: allow --range-diff to apply to a lone-patch
format-patch: add --creation-factor tweak for --range-diff
format-patch: teach --range-diff to respect -v/--reroll-count
format-patch: extend --range-diff to accept revision range
format-patch: add --range-diff option to embed diff in cover letter
range-diff: relieve callers of low-level configuration burden
range-diff: publish default creation factor
range-diff: respect diff_option.file rather than assuming 'stdout'

Merge branch 'cc/delta-islands'

Lift code from GitHub to restrict delta computation so that an
object that exists in one fork is not made into a delta against
another object that does not appear in the same forked repository.

* cc/delta-islands:
pack-objects: move 'layer' into 'struct packing_data'
pack-objects: move tree_depth into 'struct packing_data'
t5320: tests for delta islands
repack: add delta-islands support
pack-objects: add delta-islands support
pack-objects: refactor code into compute_layer_order()
Add delta-islands.{c,h}

Merge branch 'es/format-patch-interdiff'

"git format-patch" learned a new "--interdiff" option to explain
the difference between this version and the previous atttempt in
the cover letter (or after the tree-dashes as a comment).

* es/format-patch-interdiff:
format-patch: allow --interdiff to apply to a lone-patch
log-tree: show_log: make commentary block delimiting reusable
interdiff: teach show_interdiff() to indent interdiff
format-patch: teach --interdiff to respect -v/--reroll-count
format-patch: add --interdiff option to embed diff in cover letter
format-patch: allow additional generated content in make_cover_letter()

Merge branch 'jk/trailer-fixes'

"git interpret-trailers" and its underlying machinery had a buggy
code that attempted to ignore patch text after commit log message,
which triggered in various codepaths that will always get the log
message alone and never get such an input.

* jk/trailer-fixes:
append_signoff: use size_t for string offsets
sequencer: ignore "---" divider when parsing trailers
pretty, ref-filter: format %(trailers) with no_divider option
interpret-trailers: allow suppressing "---" divider
interpret-trailers: tighten check for "---" patch boundary
trailer: pass process_trailer_opts to trailer_info_get()
trailer: use size_t for iterating trailer list
trailer: use size_t for string offsets

Merge branch 'sb/range-diff-colors'

The color output support for recently introduced "range-diff"
command got tweaked a bit.

* sb/range-diff-colors:
range-diff: indent special lines as context
range-diff: make use of different output indicators
diff.c: add --output-indicator-{new, old, context}
diff.c: rewrite emit_line_0 more understandably
diff.c: omit check for line prefix in emit_line_0
diff: use emit_line_0 once per line
diff.c: add set_sign to emit_line_0
diff.c: reorder arguments for emit_line_ws_markup
diff.c: simplify caller of emit_line_0
t3206: add color test for range-diff --dual-color
test_decode_color: understand FAINT and ITALIC

Merge branch 'jk/pack-delta-reuse-with-bitmap'

When creating a thin pack, which allows objects to be made into a
delta against another object that is not in the resulting pack but
is known to be present on the receiving end, the code learned to
take advantage of the reachability bitmap; this allows the server
to send a delta against a base beyond the "boundary" commit.

* jk/pack-delta-reuse-with-bitmap:
pack-objects: reuse on-disk deltas for thin "have" objects
pack-bitmap: save "have" bitmap from walk
t/perf: add perf tests for fetches from a bitmapped server
t/perf: add infrastructure for measuring sizes
t/perf: factor out percent calculations
t/perf: factor boilerplate out of test_perf

Merge branch 'nd/unpack-trees-with-cache-tree'

The unpack_trees() API used in checking out a branch and merging
walks one or more trees along with the index. When the cache-tree
in the index tells us that we are walking a tree whose flattened
contents is known (i.e. matches a span in the index), as linearly
scanning a span in the index is much more efficient than having to
open tree objects recursively and listing their entries, the walk
can be optimized, which is done in this topic.

* nd/unpack-trees-with-cache-tree:
Document update for nd/unpack-trees-with-cache-tree
cache-tree: verify valid cache-tree in the test suite
unpack-trees: add missing cache invalidation
unpack-trees: reuse (still valid) cache-tree from src_index
unpack-trees: reduce malloc in cache-tree walk
unpack-trees: optimize walking same trees with cache-tree
unpack-trees: add performance tracing
trace.h: support nested performance tracing

Merge branch 'ds/reachable'

The code for computing history reachability has been shuffled,
obtained a bunch of new tests to cover them, and then being
improved.

* ds/reachable:
commit-reach: correct accidental #include of C file
commit-reach: use can_all_from_reach
commit-reach: make can_all_from_reach... linear
commit-reach: replace ref_newer logic
test-reach: test commit_contains
test-reach: test can_all_from_reach_with_flags
test-reach: test reduce_heads
test-reach: test get_merge_bases_many
test-reach: test is_descendant_of
test-reach: test in_merge_bases
test-reach: create new test tool for ref_newer
commit-reach: move can_all_from_reach_with_flags
upload-pack: generalize commit date cutoff
upload-pack: refactor ok_to_give_up()
upload-pack: make reachable() more generic
commit-reach: move commit_contains from ref-filter
commit-reach: move ref_newer from remote.c
commit.h: remove method declarations
commit-reach: move walk methods from commit.c

Merge branch 'sb/submodule-update-in-c'

"git submodule update" is getting rewritten piece-by-piece into C.

* sb/submodule-update-in-c:
submodule--helper: introduce new update-module-mode helper
submodule--helper: replace connect-gitdir-workingtree by ensure-core-worktree
builtin/submodule--helper: factor out method to update a single submodule
builtin/submodule--helper: store update_clone information in a struct
builtin/submodule--helper: factor out submodule updating
git-submodule.sh: rename unused variables
git-submodule.sh: align error reporting for update mode to use path

Merge branch 'tg/rerere'

Fixes to "git rerere" corner cases, especially when conflict
markers cannot be parsed in the file.

* tg/rerere:
rerere: recalculate conflict ID when unresolved conflict is committed
rerere: teach rerere to handle nested conflicts
rerere: return strbuf from handle path
rerere: factor out handle_conflict function
rerere: only return whether a path has conflicts or not
rerere: fix crash with files rerere can't handle
rerere: add documentation for conflict normalization
rerere: mark strings for translation
rerere: wrap paths in output in sq
rerere: lowercase error messages
rerere: unify error messages when read_cache fails

Merge branch 'ds/multi-pack-index'

When there are too many packfiles in a repository (which is not
recommended), looking up an object in these would require
consulting many pack .idx files; a new mechanism to have a single
file that consolidates all of these .idx files is introduced.

* ds/multi-pack-index: (32 commits)
pack-objects: consider packs in multi-pack-index
midx: test a few commands that use get_all_packs
treewide: use get_all_packs
packfile: add all_packs list
midx: fix bug that skips midx with alternates
midx: stop reporting garbage
midx: mark bad packed objects
multi-pack-index: store local property
multi-pack-index: provide more helpful usage info
midx: clear midx on repack
packfile: skip loading index if in multi-pack-index
midx: prevent duplicate packfile loads
midx: use midx in approximate_object_count
midx: use existing midx when writing new one
midx: use midx in abbreviation calculations
midx: read objects from multi-pack-index
config: create core.multiPackIndex setting
midx: write object offsets
midx: write object id fanout chunk
midx: write object ids in a chunk
...

Merge branch 'jk/branch-l-1-repurpose'

Updated plan to repurpose the "-l" option to "git branch".

* jk/branch-l-1-repurpose:
doc/git-branch: remove obsolete "-l" references
branch: make "-l" a synonym for "--list"

Merge branch 'tg/conflict-marker-size'

Developer aid.

* tg/conflict-marker-size:
.gitattributes: add conflict-marker-size for relevant files

Merge branch 'ts/doc-build-manpage-xsl-quietly'

Build tweak.

* ts/doc-build-manpage-xsl-quietly:
Documentation/Makefile: make manpage-base-url.xsl generation quieter

Merge branch 'jk/rev-list-stdin-noop-is-ok'

"git rev-list --stdin </dev/null" used to be an error; it now shows
no output without an error. "git rev-list --stdin --default HEAD"
still falls back to the given default when nothing is given on the
standard input.

* jk/rev-list-stdin-noop-is-ok:
rev-list: make empty --stdin not an error

Merge branch 'bp/checkout-new-branch-optim'

"git checkout -b newbranch [HEAD]" should not have to do as much as
checking out a commit different from HEAD. An attempt is made to
optimize this special case.

* bp/checkout-new-branch-optim:
checkout: optimize "git checkout -b <new_branch>"

Merge branch 'sg/t1404-update-ref-test-timeout'

An attempt to unflake a test a bit.

* sg/t1404-update-ref-test-timeout:
t1404: increase core.packedRefsTimeout to avoid occasional test failure

Merge branch 'nd/clone-case-smashing-warning'

Running "git clone" against a project that contain two files with
pathnames that differ only in cases on a case insensitive
filesystem would result in one of the files lost because the
underlying filesystem is incapable of holding both at the same
time. An attempt is made to detect such a case and warn.

* nd/clone-case-smashing-warning:
clone: report duplicate entries on case-insensitive filesystems

Merge branch 'mk/http-backend-content-length'

Test update.

* mk/http-backend-content-length:
http-backend test: make empty CONTENT_LENGTH test more realistic

fsck: verify multi-pack-index

When core.multiPackIndex is true, we may have a multi-pack-index
in our object directory. Add calls to 'git multi-pack-index verify'
at the end of 'git fsck' if so.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: report progress during 'verify'

When verifying a multi-pack-index, the only action that takes
significant time is checking the object offsets. For example,
to verify a multi-pack-index containing 6.2 million objects in
the Linux kernel repository takes 1.3 seconds on my machine.
99% of that time is spent looking up object offsets in each of
the packfiles and comparing them to the multi-pack-index offset.

Add a progress indicator for that section of the 'verify' verb.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify object offsets

The 'git multi-pack-index verify' command must verify the object
offsets stored in the multi-pack-index are correct. There are two
ways the offset chunk can be incorrect: the pack-int-id and the
object offset.

Replace the BUG() statement with a die() statement, now that we
may hit a bad pack-int-id during a 'verify' command on a corrupt
multi-pack-index, and it is covered by a test.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: fix 32-bit vs 64-bit size check

When loading a 64-bit offset, we intend to check that off_t can store
the resulting offset. However, the condition accidentally checks the
32-bit offset to see if it is smaller than a 64-bit value. Fix it,
and this will be covered by a test in the 'git multi-pack-index verify'
command in a later commit.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify oid lookup order

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify oid fanout order

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify missing pack

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify packname order

The final check we make while loading a multi-pack-index is that
the packfile names are in lexicographical order. Make this error
be a die() instead.

In order to test this condition, we need multiple packfiles.
Earlier in t5319-multi-pack-index.sh, we tested the interaction with
'git repack' but this limits us to one packfile in our object dir.
Move these repack tests until after the 'verify' tests.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify corrupt chunk lookup table

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: verify bad header

When verifying if a multi-pack-index file is valid, we want the
command to fail to signal an invalid file. Previously, we wrote
an error to stderr and continued as if we had no multi-pack-index.
Now, die() instead of error().

Add tests that check corrupted headers in a few ways:

* Bad signature
* Bad file version
* Bad hash version
* Truncated hash count
* Extended hash count

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

multi-pack-index: add 'verify' verb

The multi-pack-index builtin writes multi-pack-index files, and
uses a 'write' verb to do so. Add a 'verify' verb that checks this
file matches the contents of the pack-indexes it replaces.

The current implementation is a no-op, but will be extended in
small increments in later commits.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Revert "doc/Makefile: drop doc-diff worktree and temporary files on "make clean""

This reverts commit 6f924265a0bf6efa677e9a684cebdde958e5ba06, which
started to require that we have an executable git available in order
to say "make clean", which gives us a chicken-and-egg problem.

Having to have Git installed, or be in a repository, in order to be
able to run an optional "doc-diff" tool is fine. Requiring either
in order to run "make clean" is a different story.

Reported by Jonathan Nieder <jrnieder@gmail.com>.

refs: docstring typo

Signed-off-by: Tao Qingyun <taoqy@ls-a.me>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph verify: add progress output

For the reasons explained in the "commit-graph write: add progress
output" commit leading up to this one, emit progress on "commit-graph
verify". Since e0fd51e1d7 ("fsck: verify commit-graph", 2018-06-27)
"git fsck" has called this command if core.commitGraph=true, but
there's been no progress output to indicate that anything was
different. Now there is (on my tiny dotfiles.git repository):

$ git -c core.commitGraph=true -C ~/ fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (2821/2821), done.
dangling blob 5b8bbdb9b788ed90459f505b0934619c17cc605b
Verifying commits in commit graph: 100% (867/867), done.

And on a larger repository, such as the 2015-04-03-1M-git.git test
repository:

$ time git -c core.commitGraph=true -C ~/g/2015-04-03-1M-git/ commit-graph verify
Verifying commits in commit graph: 100% (1000447/1000447), done.
real 0m7.813s
[...]

Since the "commit-graph verify" subcommand is never called from "git
gc", we don't have to worry about passing some some "report_progress"
progress variable around for this codepath.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

commit-graph write: add progress output

Before this change the "commit-graph write" command didn't report any
progress. On my machine this command takes more than 10 seconds to
write the graph for linux.git, and around 1m30s on the
2015-04-03-1M-git.git[1] test repository (a test case for a large
monorepository).

Furthermore, since the gc.writeCommitGraph setting was added in
d5d5d7b641 ("gc: automatically write commit-graph files", 2018-06-27),
there was no indication at all from a "git gc" run that anything was
different. This why one of the progress bars being added here uses
start_progress() instead of start_delayed_progress(), so that it's
guaranteed to be seen. E.g. on my tiny 867 commit dotfiles.git
repository:

$ git -c gc.writeCommitGraph=true gc
Enumerating objects: 2821, done.
[...]
Computing commit graph generation numbers: 100% (867/867), done.

On larger repositories, such as linux.git the delayed progress bar(s)
will kick in, and we'll show what's going on instead of, as was
previously happening, printing nothing while we write the graph:

$ git -c gc.writeCommitGraph=true gc
[...]
Annotating commits in commit graph: 1565573, done.
Computing commit graph generation numbers: 100% (782484/782484), done.

Note that here we don't show "Finding commits for commit graph", this
is because under "git gc" we seed the search with the commit
references in the repository, and that set is too small to show any
progress, but would e.g. on a smaller repo such as git.git with
--stdin-commits:

$ git rev-list --all | git -c gc.writeCommitGraph=true write --stdin-commits
Finding commits for commit graph: 100% (162576/162576), done.
Computing commit graph generation numbers: 100% (162576/162576), done.

With --stdin-packs we don't show any estimation of how much is left to
do. This is because we might be processing more than one pack. We
could be less lazy here and show progress, either by detecting that
we're only processing one pack, or by first looping over the packs to
discover how many commits they have. I don't see the point in doing
that work. So instead we get (on 2015-04-03-1M-git.git):

$ echo pack-<HASH>.idx | git -c gc.writeCommitGraph=true --exec-path=$PWD commit-graph write --stdin-packs
Finding commits for commit graph: 13064614, done.
Annotating commits in commit graph: 3001341, done.
Computing commit graph generation numbers: 100% (1000447/1000447), done.

No GC mode uses --stdin-packs. It's what they use at Microsoft to
manually compute the generation numbers for their collection of large
packs which are never coalesced.

The reason we need a "report_progress" variable passed down from "git
gc" is so that we don't report this output when we're running in the
process "git gc --auto" detaches from the terminal.

Since we write the commit graph from the "git gc" process itself (as
opposed to what we do with say the "git repack" phase), we'd end up
writing the output to .git/gc.log and reporting it to the user next
time as part of the "The last gc run reported the following[...]"
error, see 329e6e8794 ("gc: save log from daemonized gc --auto and
print it next time", 2015-09-19).

So we must keep track of whether or not we're running in that
demonized mode, and if so print no progress.

See [2] and subsequent replies for a discussion of an approach not
taken in compute_generation_numbers(). I.e. we're saying "Computing
commit graph generation numbers", even though on an established
history we're mostly skipping over all the work we did in the
past. This is similar to the white lie we tell in the "Writing
objects" phase (not all are objects being written).

Always showing progress is considered more important than
accuracy. I.e. on a repository like 2015-04-03-1M-git.git we'd hang
for 6 seconds with no output on the second "git gc" if no changes were
made to any objects in the interim if we'd take the approach in [2].

1. https://github.com/avar/2015-04-03-1M-git

2. <c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com>
(https://public-inbox.org/git/c6960252-c095-fb2b-e0bc-b1e6bb261614@gmail.com/)

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t0014: introduce an alias testing suite

Introduce a testing suite that is dedicated to aliases.
For now, check only if nested aliases work and if looping
aliases are detected successfully.

The looping aliases check for mixed execution is there but
disabled, because it is blocking the test suite for a full
minute. As soon as there is a solution for loops using
external commands, it should be enabled.

Signed-off-by: Tim Schumacher <timschumi@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

alias: show the call history when an alias is looping

Just printing the command that the user entered is not particularly
helpful when trying to find the alias that causes the loop.

Print the history of substituted commands to help the user find the
offending alias. Mark the entrypoint of the loop with "<==" and the
last command (which looped back to the entrypoint) with "==>".

Signed-off-by: Tim Schumacher <timschumi@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

alias: add support for aliases of an alias

Aliases can only contain non-alias git commands and their arguments,
not other user-defined aliases. Resolving further (nested) aliases
is prevented by breaking the loop after the first alias was
processed. Git then fails with a command-not-found error.

Allow resolving nested aliases by not breaking the loop in
run_argv() after the first alias was processed. Instead, continue
the loop until `handle_alias()` fails, which means that there are no
further aliases that can be processed. Prevent looping aliases by
storing substituted commands in `cmd_list` and checking if a command
has been substituted previously.

While we're at it, fix a styling issue just below the added code.

Signed-off-by: Tim Schumacher <timschumi@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t5318: use test_oid for HASH_LEN

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1407: make hash size independent

Instead of hard-coding a 40-based constant, split the output of
for-each-ref and for-each-reflog by field.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1406: make hash-size independent

Instead of hard-coding a 40-based constant, split the output of
for-each-ref and for-each-reflog by field.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1405: make hash size independent

Instead of hard-coding a 40-based constant, split the output of
for-each-ref and for-each-reflog by field.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1400: switch hard-coded object ID to variable

Switch a hard-coded all-zeros object ID to use a variable instead.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t1006: make hash size independent

Compute the size of the tree and commit objects we're creating by
checking for the size of an object ID and computing the resulting sizes
accordingly.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t0064: make hash size independent

Compute test values of the appropriate size instead of hard-coding
40-character values. Rename the echo20 function to echoid, since the
values may be of varying sizes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

builtin/remote: quote remote name on error to display empty name

When adding new remote name with empty string, git will print the
following error message,

fatal: '' is not a valid remote name\n

But when removing remote name with empty string as input, git shows the
empty string without quote,

fatal: No such remote: \n

To make these error messages consistent, quote the name of the remote
that we tried and failed to find.

Signed-off-by: Shulhan <m.shulhan@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

linear-assignment: fix potential out of bounds memory access

Currently the 'compute_assignment()' function may read memory out
of bounds, even if used correctly. Namely this happens when we only
have one column. In that case we try to calculate the initial
minimum cost using '!j1' as column in the reduction transfer code.
That in turn causes us to try and get the cost from column 1 in the
cost matrix, which does not exist, and thus results in an out of
bounds memory read.

In the original paper [1], the example code initializes that minimum
cost to "infinite". We could emulate something similar by setting the
minimum cost to INT_MAX, which would result in the same minimum cost
as the current algorithm, as we'd always go into the if condition at
least once, except when we only have one column, and column_count thus
equals 1.

If column_count does equal 1, the condition in the loop would always
be false, and we'd end up with a minimum of INT_MAX, which may lead to
integer overflows later in the algorithm.

For a column count of 1, we however do not even really need to go
through the whole algorithm. A column count of 1 means that there's
no possible assignments, and we can just zero out the column2row and
row2column arrays, and return early from the function, while keeping
the reduction transfer part of the function the same as it is
currently.

Another solution would be to just not call the 'compute_assignment()'
function from the range diff code in this case, however it's better to
make the compute_assignment function more robust, so future callers
don't run into this potential problem.

Note that the test only fails under valgrind on Linux, but the same
command has been reported to segfault on Mac OS.

[1]: Jonker, R., & Volgenant, A. (1987). A shortest augmenting path
algorithm for dense and sparse linear assignment
problems. Computing, 38(4), 325–340.

Reported-by: ryenus <ryenus@gmail.com>
Helped-by: Derrick Stolee <stolee@gmail.com>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t0002: abstract away SHA-1 specific constants

Adjust the test so that it computes variables for object IDs instead of
using hard-coded hashes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t0000: update tests for SHA-256

Test t0000 tests the "basics of the basics" and as such, checks that we
have various fixed hard-coded object IDs. The tests relying on these
assertions have been marked with the SHA1 prerequisite, as they will
obviously not function in their current form with SHA-256.

Use the test_oid helper to update these assertions and provide values
for both SHA-1 and SHA-256.

These object IDs were synthesized using a set of scripts that created
the objects for both SHA-1 and SHA-256 using the same method to ensure
that they are indeed the correct values.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t0000: use hash translation table

If the hash we're using is 32 bytes in size, attempting to insert a
20-byte object name won't work. Since these are synthesized objects
that are almost all zeros, look them up in a translation table.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

t: add test functions to translate hash-related values

Add several test functions to make working with various hash-related
values easier.

Add test_oid_init, which loads common hash-related constants and
placeholder object IDs from the newly added files in t/oid-info.
Provide values for these constants for both SHA-1 and SHA-256.

Add test_oid_cache, which accepts data on standard input in the form of
hash-specific key-value pairs that can be looked up later, using the
same format as the files in t/oid-info. Document this format in a
t/oid-info/README directory so that it's easier to use in the future.

Add test_oid, which is used to specify look up a per-hash value
(produced on standard output) based on the key specified as its
argument. Usually the data to be looked up will be a hash-related
constant (such as the size of the hash in binary or hexadecimal), a
well-known or placeholder object ID (such as the all-zeros object ID or
one consisting of "deadbeef" repeated), or something similar. For these
reasons, test_oid will usually be used within a command substitution.
Consequently, redirect the error output to standard error, since
otherwise it will not be displayed.

Add test_detect_hash, which currently only detects SHA-1, and
test_set_hash, which can be used to set a different hash algorithm for
test purposes. In the future, test_detect_hash will learn to actually
detect the hash depending on how the testsuite is to be run.

Use the local keyword within these functions to avoid overwriting other
shell variables. We have had a test balloon in place for a couple of
releases to catch shells that don't have this keyword and have not
received any reports of failure. Note that the varying usages of local
used here are supported by all common open-source shells supporting the
local keyword.

Test these new functions as part of t0000, which also serves to
demonstrate basic usage of them. In addition, add documentation on how
to format the lookup data and how to use the test functions.

Implement two basic lookup charts, one for common invalid or synthesized
object IDs, and one for various facts about the hash function in use.
Provide versions of the data for both SHA-1 and SHA-256.

Since we use shell variables for storage, names used for lookup can
currently consist only of shell identifier characters. If this is a
problem in the future, we can hash the names before use.

Improved-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fetch-object: set exact_oid when fetching

fetch_objects() currently does not set exact_oid in struct ref when
invoking transport_fetch_refs(). If the server supports ref-in-want,
fetch_pack() uses this field to determine whether a wanted ref should be
requested as a "want-ref" line or a "want" line; without the setting of
exact_oid, the wrong line will be sent.

Set exact_oid, so that the correct line is sent.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fetch-object: unify fetch_object[s] functions

There are fetch_object() and fetch_objects() helpers in
fetch-object.h; as the latter takes "struct oid_array",
the former cannot be made into a thin wrapper around the
latter without an extra allocation and set-up cost.

Update fetch_objects() to take an array of "struct object_id"
and number of elements in it as separate parameters, remove
fetch_object(), and adjust all existing callers of these
functions to use the new fetch_objects().

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

sequencer: fix --allow-empty-message behavior, make it smarter

In commit b00bf1c9a8dd ("git-rebase: make --allow-empty-message the
default", 2018-06-27), several arguments were given for transplanting
empty commits without halting and asking the user for confirmation on
each commit. These arguments were incomplete because the logic clearly
assumed the only cases under consideration were transplanting of commits
with empty messages (see the comment about "There are two sources for
commits with empty messages). It didn't discuss or even consider
rewords, squashes, etc. where the user is explicitly asked for a new
commit message and provides an empty one. (My bad, I totally should
have thought about that at the time, but just didn't.)

Rewords and squashes are significantly different, though, as described
by SZEDER:

Let's suppose you start an interactive rebase, choose a commit to
squash, save the instruction sheet, rebase fires up your editor, and
then you notice that you mistakenly chose the wrong commit to
squash. What do you do, how do you abort?

Before [that commit] you could clear the commit message, exit the
editor, and then rebase would say "Aborting commit due to empty
commit message.", and you get to run 'git rebase --abort', and start
over.

But [since that commit, ...] saving the commit message as is would
let rebase continue and create a bunch of unnecessary objects, and
then you would have to use the reflog to return to the pre-rebase
state.

Also, he states:

The instructions in the commit message template, which is shown for
'reword' and 'squash', too, still say...

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.

These are sound arguments that when editing commit messages during a
sequencer operation, that if the commit message is empty then the
operation should halt and ask the user to correct. The arguments in
commit b00bf1c9a8dd (referenced above) still apply when transplanting
previously created commits with empty commit messages, so the sequencer
should not halt for those.

Furthermore, all rationale so far applies equally for cherry-pick as for
rebase. Therefore, make the code default to --allow-empty-message when
transplanting an existing commit, and to default to halting when the
user is asked to edit a commit message and provides an empty one -- for
both rebase and cherry-pick.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: support comments & empty lines in skipList

It's annoying not to be able to put comments and empty lines in the
skipList, when e.g. keeping a big central list of commits to skip in
/etc/gitconfig, which was my motivation for 1362df0d41 ("fetch:
implement fetch.fsck.*", 2018-07-27).

Implement that, and document what version of Git this was changed in,
since this on-disk format can be expected to be used by multiple
versions of git.

There is no notable performance impact from this change, using the
test setup described a couple of commits back:

Test HEAD~ HEAD
----------------------------------------------------------------------------------------
1450.3: fsck with 0 skipped bad commits 7.69(7.27+0.42) 7.86(7.48+0.37) +2.2%
1450.5: fsck with 1 skipped bad commits 7.69(7.30+0.38) 7.83(7.47+0.36) +1.8%
1450.7: fsck with 10 skipped bad commits 7.76(7.38+0.38) 7.79(7.38+0.41) +0.4%
1450.9: fsck with 100 skipped bad commits 7.76(7.38+0.38) 7.74(7.36+0.38) -0.3%
1450.11: fsck with 1000 skipped bad commits 7.71(7.30+0.41) 7.72(7.34+0.38) +0.1%
1450.13: fsck with 10000 skipped bad commits 7.74(7.34+0.40) 7.72(7.34+0.38) -0.3%
1450.15: fsck with 100000 skipped bad commits 7.75(7.40+0.35) 7.70(7.29+0.40) -0.6%
1450.17: fsck with 1000000 skipped bad commits 7.12(6.86+0.26) 7.13(6.87+0.26) +0.1%

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: use oidset instead of oid_array for skipList

Change the implementation of the skipList feature to use oidset
instead of oid_array to store SHA-1s for later lookup.

This list is parsed once on startup by fsck, fetch-pack or
receive-pack depending on the *.skipList config in use. I.e. only once
per invocation, but note that for "clone --recurse-submodules" each
submodule will re-parse the list, in addition to the main project, and
it will be re-parsed when checking .gitmodules blobs, see
fb16287719 ("fsck: check skiplist for object in fsck_blob()",
2018-06-27).

Memory usage is a bit higher, but we don't need to keep track of the
sort order anymore. Embed the oidset into struct fsck_options to make
its ownership clear (no hidden sharing) and avoid unnecessary pointer
indirection.

The cumulative impact on performance of this & the preceding change,
using the test setup described in the previous commit:

Test HEAD~2 HEAD~ HEAD
----------------------------------------------------------------------------------------------------------------
1450.3: fsck with 0 skipped bad commits 7.70(7.31+0.38) 7.72(7.33+0.38) +0.3% 7.70(7.30+0.40) +0.0%
1450.5: fsck with 1 skipped bad commits 7.84(7.47+0.37) 7.69(7.32+0.36) -1.9% 7.71(7.29+0.41) -1.7%
1450.7: fsck with 10 skipped bad commits 7.81(7.40+0.40) 7.94(7.57+0.36) +1.7% 7.92(7.55+0.37) +1.4%
1450.9: fsck with 100 skipped bad commits 7.81(7.42+0.38) 7.95(7.53+0.41) +1.8% 7.83(7.42+0.41) +0.3%
1450.11: fsck with 1000 skipped bad commits 7.99(7.62+0.36) 7.90(7.50+0.40) -1.1% 7.86(7.49+0.37) -1.6%
1450.13: fsck with 10000 skipped bad commits 7.98(7.57+0.40) 7.94(7.53+0.40) -0.5% 7.90(7.45+0.44) -1.0%
1450.15: fsck with 100000 skipped bad commits 7.97(7.57+0.39) 8.03(7.67+0.36) +0.8% 7.84(7.43+0.41) -1.6%
1450.17: fsck with 1000000 skipped bad commits 7.72(7.22+0.50) 7.28(7.07+0.20) -5.7% 7.13(6.87+0.25) -7.6%

Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: use strbuf_getline() to read skiplist file

The buffer is unlikely to contain a NUL character, so printing its
contents using %s in a die() format is unsafe (detected with ASan).

Use an idiomatic strbuf_getline() loop instead, which ensures the buffer
is always NUL-terminated, supports CRLF files as well, accepts files
without a newline after the last line, supports any hash length
automatically, and is shorter.

This fixes a bug where emitting an error about an invalid line on say
line 1 would continue printing subsequent lines, and usually continue
into uninitialized memory.

The performance impact of this, on a CentOS 7 box with RedHat GCC
4.8.5-28:

$ GIT_PERF_REPEAT_COUNT=5 GIT_PERF_MAKE_OPTS='-j56 CFLAGS="-O3"' ./run HEAD~ HEAD p1451-fsck-skip-list.sh
Test HEAD~ HEAD
----------------------------------------------------------------------------------------
1450.3: fsck with 0 skipped bad commits 7.75(7.39+0.35) 7.68(7.29+0.39) -0.9%
1450.5: fsck with 1 skipped bad commits 7.70(7.30+0.40) 7.80(7.42+0.37) +1.3%
1450.7: fsck with 10 skipped bad commits 7.77(7.37+0.40) 7.87(7.47+0.40) +1.3%
1450.9: fsck with 100 skipped bad commits 7.82(7.41+0.40) 7.88(7.43+0.44) +0.8%
1450.11: fsck with 1000 skipped bad commits 7.88(7.49+0.39) 7.84(7.43+0.40) -0.5%
1450.13: fsck with 10000 skipped bad commits 8.02(7.63+0.39) 8.07(7.67+0.39) +0.6%
1450.15: fsck with 100000 skipped bad commits 8.01(7.60+0.41) 8.08(7.70+0.38) +0.9%
1450.17: fsck with 1000000 skipped bad commits 7.60(7.10+0.50) 7.37(7.18+0.19) -3.0%

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: add a performance test for skipList

Create a performance test to see how the skipList implementation
performs. First we setup N bad commits, then we see how progressively
working our way up to 0..N in increments of 10x does. I.e. the
needle(s) in the haystack get progressively more numerous.

Signed-off-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: add a performance test

Add a plain performance test for "fsck". This test will not be used to
/ referred to in any upcoming commit of mine in this series, but
having a simple test for fsck performance is valuable, so let's add it
while we're at it.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: document that skipList input must be unabbreviated

Abbreviating the SHA-1s in the skipList input has never worked, but
the documentation hasn't unambiguously stated that this is an error,
and there was no test for it.

Let's fix both since it would be easy for some later refactoring
e.g. switch to accidentally switch to a looser OID parsing function,
causing the tests before this change to pass, but for older versions
of git to be incompatible with the new skipList format.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: document and test commented & empty line skipList input

There is currently no comment syntax for the fsck.skipList, this isn't
really by design, and it would be nice to have support for comments.

Document that this doesn't work, and test for how this errors
out. These tests reveal a current bug, if there's invalid input the
output will emit some of the next line, and then go into uninitialized
memory. This is fixed in a subsequent change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck: document and test sorted skipList input

Ever since the skipList support was first added in cd94c6f91 ("fsck:
git receive-pack: support excluding objects from fsck'ing",
2015-06-22) the documentation for the format has that the file is a
sorted list of object names.

Thus, anyone using the feature would have thought the list needed to
be sorted. E.g. I recently in conjunction with my fetch.fsck.*
implementation in 1362df0d41 ("fetch: implement fetch.fsck.*",
2018-07-27) wrote some code to ship a skipList, and went out of my way
to sort it.

Doing so seems intuitive, since it contains fixed-width records, and
has no support for comments, so one might expect it to be binary
searched in-place on-disk.

However, as documented here this was never a requirement, so let's
change the documentation. Since this is a file format change let's
also document what was said about this in the past, so e.g. someone
like myself reading the new docs can see this never needed to be
sorted ("why do I have all this code to sort this thing...").

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck tests: add a test for no skipList input

The recent 65a836fa6b ("fsck: add stress tests for fsck.skipList",
2018-07-27) added various stress tests for odd invocations of
fsck.skipList, but didn't tests for some very simple ones, such as
asserting that providing to skipList with a bad commit causes fsck to
exit with a non-zero exit code. Add such a test.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

fsck tests: setup of bogus commit object

Several fsck tests used the exact same git-hash-object output, but had
copy/pasted that part of the setup code. Let's instead do that setup
once and use it in subsequent tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

update-ref: allow --no-deref with --stdin

If passed both --no-deref and --stdin, update-ref would error out with a
general usage message that did not at all suggest these options were
incompatible. The manpage for update-ref did suggest through its
synopsis line that --no-deref and --stdin were incompatible, but it sadly
also incorrectly suggested that -d and --no-deref were incompatible. So
the help around the --no-deref option is buggy in a few ways.

The --stdin option did provide a different mechanism for avoiding
dereferencing symbolic-refs: adding a line reading
option no-deref
before every other directive in the input. (Technically, if the user
wants to do the extra work of first determining which refs they want to
update or delete are symbolic, then they only need to put the extra
"option no-deref" lines before the updates of those refs. But in some
cases, that's more work than just adding the "option no-deref" before
every other directive.)

It's easier to allow the user to just pass --no-deref along with --stdin
in order to tell update-ref that the user doesn't want any symbolic ref
to be dereferenced. It also makes the update-ref documentation simpler.
Implement that, and update the documentation to match.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

update-ref: fix type of update_flags variable to match its usage

The ref_transaction_*() family of functions expect a flags parameter
which is of type unsigned int. Make the update_flags variable, which
is passed as that parameter, be of the same type.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

Make git_check_attr() a void function

git_check_attr() returns always 0.
Remove all the error handling code of the callers, which is never executed.
Change git_check_attr() to be a void function.

Signed-off-by: Torsten Bögershausen <tboegi@web.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>