* jk/prune-packed-server-info:
repack: call prune_packed_objects() and update_server_info() directly
server-info: clean up after writing info/packs
make update-server-info more robust
prune-packed: fix minor memory leak
"hash-object" learned a new "--literally" option to hash any random
garbage into a loose object, to allow us to create a test data for
mechanisms to catch corrupt objects.
* jc/hash-object:
hash-object: add --literally option
hash-object: pass 'write_object' as a flag
hash-object: reduce file-scope statics
Teach "git fsck" to inspect the contents of annotated tag objects.
* js/fsck-tag-validation:
Make sure that index-pack --strict checks tag objects
Add regression tests for stricter tag fsck'ing
fsck: check tag objects' headers
Make sure fsck_commit_buffer() does not run out of the buffer
fsck_object(): allow passing object data separately from the object itself
Refactor type_from_string() to allow continuing after detecting an error
Optimize the check to see if a ref $F can be created by making sure
no existing ref has $F/ as its prefix, which especially matters in
a repository with a large number of existing refs.
* jk/faster-name-conflicts:
refs: speed up is_refname_available
Merge branch 'jn/unpack-trees-checkout-m-carry-deletion' into maint
* jn/unpack-trees-checkout-m-carry-deletion:
checkout -m: attempt merge when deletion of path was staged
unpack-trees: use 'cuddled' style for if-else cascade
unpack-trees: simplify 'all other failures' case
* jc/apply-ws-prefix:
apply: omit ws check for excluded paths
apply: hoist use_patch() helper for path exclusion up
apply: use the right attribute for paths in non-Git patches
* jk/pretty-empty-format:
pretty: make empty userformats truly empty
pretty: treat "--format=" as an empty userformat
revision: drop useless string offset when parsing "--pretty"
* tb/crlf-tests:
MinGW: update tests to handle a native eol of crlf
Makefile: propagate NATIVE_CRLF to C
t0027: Tests for core.eol=native, eol=lf, eol=crlf
* rs/more-uses-of-skip-prefix:
pack-write: simplify index_pack_lockfile using skip_prefix() and xstrfmt()
connect: simplify check_ref() using skip_prefix() and starts_with()
A broken reimplementation of Git could write an invalid index that
records both stage #0 and higher stage entries for the same path.
Notice and reject such an index, as there is no sensible fallback
(we do not know if the broken tool wanted to resolve and forgot to
remove higher stage entries, or if it wanted to unresolve and
forgot to remove the stage#0 entry).
* jp/index-with-corrupt-stages:
read_index_unmerged(): remove unnecessary loop index adjustment
read_index_from(): catch out of order entries when reading an index file
When receiving an invalid pack stream that records the same object
twice, multiple threads got confused due to a race. We should
reject or correct such a stream upon receiving, but that will be a
larger change.
* jk/index-pack-threading-races:
index-pack: fix race condition with duplicate bases
* jk/commit-author-parsing:
determine_author_info(): copy getenv output
determine_author_info(): reuse parsing functions
date: use strbufs in date-formatting functions
record_author_date(): use find_commit_header()
record_author_date(): fix memory leak on malformed commit
commit: provide a function to find a header in a buffer
"log --date=iso" uses a slight variant of ISO 8601 format that is
made more human readable. A new "--date=iso-strict" option gives
datetime output that is more strictly conformant.
* bb/date-iso-strict:
pretty: provide a strict ISO 8601 date format
Sometimes users want to report a bug they experience on their
repository, but they are not at liberty to share the contents of
the repository. "fast-export" was taught an "--anonymize" option
to replace blob contents, names of people and paths and log
messages with bland and simple strings to help them.
* jk/fast-export-anonymize:
docs/fast-export: explain --anonymize more completely
teach fast-export an --anonymize option
The number of refs that can be pushed at once over smart HTTP was
limited by the command line length. The limitation has been lifted
by passing these refs from the standard input of send-pack.
* jk/send-pack-many-refspecs:
send-pack: take refspecs over stdin
branch: clean up commit flags after merge-filter walk
When we run `branch --merged`, we use prepare_revision_walk
with the merge-filter marked as UNINTERESTING. Any branch
tips that are marked UNINTERESTING after it returns must be
ancestors of that commit. As we iterate through the list of
refs to show, we check item->commit->object.flags to see
whether it was marked.
This interacts badly with --verbose, which will do a
separate walk to find the ahead/behind information for each
branch. There are two bad things that can happen:
1. The ahead/behind walk may get the wrong results,
because it can see a bogus UNINTERESTING flag leftover
from the merge-filter walk.
2. We may omit some branches if their tips are involved in
the ahead/behind traversal of a branch shown earlier.
The ahead/behind walk carefully cleans up its commit
flags, meaning it may also erase the UNINTERESTING
flag that we expect to check later.
We can solve this by moving the merge-filter state for each
ref into its "struct ref_item" as soon as we finish the
merge-filter walk. That fixes (2). Then we are free to clear
the commit flags we used in the walk, fixing (1).
Note that we actually do away with the matches_merge_filter
helper entirely here, and inline it between the revision
walk and the flag-clearing. This ensures that nobody
accidentally calls it at the wrong time (it is only safe to
check in that instant between the setting and clearing of
the global flag).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The macro ALLOC_GROW manages several aspects of dynamic memory
allocations for arrays: It performs overprovisioning in order to avoid
reallocations in future calls, updates the allocation size variable,
multiplies the item size and thus allows users to simply specify the
item count, performs the reallocation and updates the array pointer.
Sometimes this is too much. Add the macro REALLOC_ARRAY, which only
takes care of the latter three points and allows users to specfiy the
number of items the array can store. It can increase and also decrease
the size. Using the macro avoid duplicating the variable name and
takes care of the item sizes automatically.
Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
receive-pack: allow hooks to ignore its standard input stream
The pre-receive and post-receive hooks were designed to be an
improvement over old style update and post-update hooks, which take
the update information on their command line and are limited by the
command line length limit. The same information is fed from the
standard input to pre/post-receive hooks instead to lift this
limitation. It has been mandatory for these new style hooks to
consume the update information fully from the standard input stream.
Otherwise, they would risk killing the receive-pack process via
SIGPIPE.
If a hook does not want to look at all the information, it is easy
to send its standard input to /dev/null (perhaps a niche use of hook
might need to know only the fact that a push was made, without
having to know what objects have been pushed to update which refs),
and this has already been done by existing hooks that are written
carefully.
However, because there is no good way to consistently fail hooks
that do not consume the input fully (a small push may result in a
short update record that may fit within the pipe buffer, to which
the receive-pack process may manage to write before the hook has a
chance to exit without reading anything, which will not result in a
death-by-SIGPIPE of receive-pack), it can lead to a hard to diagnose
"once in a blue moon" phantom failure.
Lift this "hooks must consume their input fully" mandate. A mandate
that is not enforced strictly is not helping us to catch mistakes in
hooks. If a hook has a good reason to decide the outcome of its
operation without reading the information we feed it, let it do so
as it pleases.
Documentation/git-rebase.txt: <upstream> must be given to specify <branch>
Current syntax description makes one wonder if there is any
syntactic way to distinguish between <branch> and <upstream> so that
one can specify <branch> but not <upstream>, but that is not the
case.
Make it explicit that these arguments are positional, i.e. the
earlier ones cannot be omitted if you want to give later ones.
Signed-off-by: Sergey Organov <sorganov@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If the stderr of "git credential-cache" is redirected to a
pipe, the reader on the other end of a pipe may be surprised
that the pipe remains open long after the process exits.
This happens because we may auto-spawn a daemon which is
long-lived, and which keeps stderr open.
We can solve this by redirecting the daemon's stderr to
/dev/null once we are ready to go into our event loop. We
would not want to do so before then, because we may want to
report errors about the setup (e.g., failure to establish
the listening socket).
This does mean that we will not report errors we encounter
for specific clients. That's acceptable, as such errors
should be rare (e.g., clients sending buggy requests).
However, we also provide an escape hatch: if you want to see
these later messages, you can provide the "--debug" option
to keep stderr open.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
We allocate pack information in a static global list but
never clean it up. This leaks memory, and means that calling
update_server_info twice will generate a buggy file (it will
have duplicate entries).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Since "git update-server-info" may be called automatically
as part of a push or a "gc --auto", we should be robust
against two processes trying to update it simultaneously.
However, we currently use a fixed tempfile, which means that
two simultaneous writers may step on each other's toes and
end up renaming junk into place.
Let's instead switch to using a unique tempfile via mkstemp.
We do not want to use a lockfile here, because it's OK for
two writers to simultaneously update (one will "win" the
rename race, but that's OK; they should be writing the same
information).
While we're there, let's clean up a few other things:
1. Detect write errors. Report them and abort the update
if any are found.
2. Free path memory rather than leaking it (and clean up
the tempfile when necessary).
3. Use the pathdup functions consistently rather than
static buffers or manually calculated lengths.
This last one fixes a potential overflow of "infofile" in
update_info_packs (e.g., by putting large junk into
$GIT_OBJECT_DIRECTORY). However, this overflow was probably
not an interesting attack vector for two reasons:
a. The attacker would need to control the environment to
do this, in which case it was already game-over.
b. During its setup phase, git checks that the directory
actually exists, which means it is probably shorter
than PATH_MAX anyway.
Because both update_info_refs and update_info_packs share
these same failings (and largely duplicate each other), this
patch factors out the improved error-checking version into a
helper function.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
On my Debian 7 system, this fixes annoying warnings when the output
of "git svn" commands are redirected:
Unable to get Terminal Size. The TIOCGWINSZ ioctl didn't work.
The COLUMNS and LINES environment variables didn't work. The
resize program didn't work.
git svn: find-rev allows short switches for near matches
Allow -B and -A to act as short aliases for --before and --after
options respectively. This reduces typing and hopefully allows
reuse of muscle memory for grep(1) users.
Git no longer seems to use these flags or their associated config keys;
when they are present, git-svn outputs a message indicating that they
are being ignored.
Signed-off-by: Lawrence Velázquez <vq@larryv.me> Signed-off-by: Eric Wong <normalperson@yhbt.net>
Calling "git svn info $(pwd)" would hit:
"Reading from filehandle failed at ..."
errors due to improper prefixing and canonicalization.
Strip the toplevel path from absolute filesystem paths to ensure
downstream canonicalization routines are only exposed to paths
tracked in git (or SVN).
v2:
Thanks to Andrej Manduch for originally noticing the issue
and fixing my original version of this to handle
more corner cases such as "/path/to/top/../top" and
"/path/to/top/../top/file" as shown in the new test cases.
v3:
Fix pathname portability problems pointed out by Johannes Sixt
with a hint from brian m. carlson.
Cc: Johannes Sixt <j6t@kdbg.org> Cc: "brian m. carlson" <sandals@crustytoothpaste.net> Signed-off-by: Andrej Manduch <amanduch@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: branch: avoid systematic prompt for cert/pass
Commands such as "git svn init/fetch/dcommit" do not prompt for client
certificate/password if they are stored in SVN config file. Make
"git svn branch" consistent with the other commands, as SVN::Client is
capable of building its own authentication baton from information in the
SVN config directory.
Signed-off-by: Monard Vong <travelingsoul86@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
t9300: use test_cmp_bin instead of test_cmp to compare binary files
test_cmp is intended to produce diff output for human consumption. The
input in one instance in t9300-fast-import.sh are binary files, however.
Use test_cmp_bin to compare the files.
This was noticed because on Windows we have a special implementation of
test_cmp in pure bash code (to ignore differences due to intermittent CR
in actual output), and bash runs into an infinite loop due to the binary
nature of the input.
Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Our filesystem ref storage does not allow D/F conflicts; so
if "refs/heads/a/b" exists, we do not allow "refs/heads/a"
to exist (and vice versa). This falls out naturally for
loose refs, where the filesystem enforces the condition. But
for packed-refs, we have to make the check ourselves.
We do so by iterating over the entire packed-refs namespace
and checking whether each name creates a conflict. If you
have a very large number of refs, this is quite inefficient,
as you end up doing a large number of comparisons with
uninteresting bits of the ref tree (e.g., we know that all
of "refs/tags" is uninteresting in the example above, yet we
check each entry in it).
Instead, let's take advantage of the fact that we have the
packed refs stored as a trie of ref_entry structs. We can
find each component of the proposed refname as we walk
through the trie, checking for D/F conflicts as we go. For a
refname of depth N (i.e., 4 in the above example), we only
have to visit N nodes. And at each visit, we can binary
search the M names at that level, for a total complexity of
O(N lg M). ("M" is different at each level, of course, but
we can take the worst-case "M" as a bound).
In a pathological case of fetching 30,000 fresh refs into a
repository with 8.5 million refs, this dropped the time to
run "git fetch" from tens of minutes to ~30s.
This may also help smaller cases in which we check against
loose refs (which we do when renaming a ref), as we may
avoid a disk access for unrelated loose directories.
Note that the tests we add appear at first glance to be
redundant with what is already in t3210. However, the early
tests are not robust; they are run with reflogs turned on,
meaning that we are not actually testing
is_refname_available at all! The operations will still fail
because the reflogs will hit D/F conflicts in the
filesystem. To get a true test, we must turn off reflogs
(but we don't want to do so for the entire script, because
the point of turning them on was to cover some other cases).
Reviewed-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
t1450: make sure fsck detects a malformed tagger line
With "hash-object --literally", write a tag object that is not
supposed to pass one of the new checks added to "fsck", and make
sure that the new check catches the breakage.
* js/fsck-tag-validation:
Make sure that index-pack --strict checks tag objects
Add regression tests for stricter tag fsck'ing
fsck: check tag objects' headers
Make sure fsck_commit_buffer() does not run out of the buffer
fsck_object(): allow passing object data separately from the object itself
Refactor type_from_string() to allow continuing after detecting an error
Make sure that index-pack --strict checks tag objects
One of the most important use cases for the strict tag object checking
is when transfer.fsckobjects is set to true to catch invalid objects
early on. This new regression test essentially tests the same code path
by directly calling 'index-pack --strict' on a pack containing an
tag object without a 'tagger' line.
Technically, this test is not enough: it only exercises a code path that
*warns*, not one that *fails*. The reason is that hash-object and
pack-objects both insist on parsing the tag objects and would fail on
invalid tag objects at this time.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Fsck tries hard to detect missing objects, and will complain
(and exit non-zero) about any inter-object links that are
missing. However, it will not exit non-zero for any missing
ref tips, meaning that a severely broken repository may
still pass "git fsck && echo ok".
The problem is that we use for_each_ref to iterate over the
ref tips, which hides broken tips. It does at least print an
error from the refs.c code, but fsck does not ever see the
ref and cannot note the problem in its exit code. We can solve
this by using for_each_rawref and noting the error ourselves.
In addition to adding tests for this case, we add tests for
all types of missing-object links (all of which worked, but
which we were not testing).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Introduce CONFIG_REGEX_NONE as a more explicit sentinel value to say
"we do not want to replace any existing entry" and use it in the
implementation of "git config --add".
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This allows "hash-object --stdin" to just hash any garbage into a
"loose object" that may not pass the standard object parsing check
or fsck, so that different kind of corrupt objects we may encounter
in the field can be imitated in our test suite. That would in turn
allow us to test features that catch these corrupt objects.
Note that "cat-file" may need to learn "--literally" option to allow
us peek into a truly broken object.
The intent of the new test case is to catch general breakages in
the fsck_tag() function, not so much to test it extensively, trying to
strike the proper balance between thoroughness and speed.
While it *would* have been nice to test the code path where fsck_object()
encounters an invalid tag object, this is not possible using git fsck: tag
objects are parsed already before fsck'ing (and the parser already fails
upon such objects).
Even worse: we would not even be able write out invalid tag objects
because git hash-object parses those objects, too, unless we resorted to
really ugly hacks such as using something like this in the unit tests
(essentially depending on Perl *and* Compress::Zlib):
Instead of forcing callers of lower level functions write
(write_object ? HASH_WRITE_OBJECT : 0), prepare the flag to be
passed down in the callchain from the command line parser.
Most of the knobs that affect helper functions called from
cmd_hash_object() were passed to them as parameters already, and the
only effect of having them as file-scope statics was to make the
reader wonder if the parameters are hiding the file-scope global
values by accident. Adjust their initialisation and make them
function-local variables.
The only exception was no_filters hash_stdin_paths() peeked from the
file-scope global, which was converted to a parameter to the helper
function.
Make sure fsck_commit_buffer() does not run out of the buffer
So far, we assumed that the buffer is NUL terminated, but this is not
a safe assumption, now that we opened the fsck_object() API to pass a
buffer directly.
So let's make sure that there is at least an empty line in the buffer.
That way, our checks would fail if the empty line was encountered
prematurely, and consequently we can get away with the current string
comparisons even with non-NUL-terminated buffers are passed to
fsck_object().
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Documentation: use single-parameter --cacheinfo in example
The single-parameter form is described as the preferred way. Separate
arguments are only supported for backward compatibility. Update the
example to the recommended form.
Signed-off-by: Steffen Prohaska <prohaska@zib.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The API to allocate the structure to keep track of commit
decoration was cumbersome to use, inviting lazy code to
overallocate memory.
* jk/name-decoration-alloc:
log-tree: use FLEX_ARRAY in name_decoration
log-tree: make name_decoration hash static
log-tree: make add_name_decoration a public function
"git checkout -m" did not switch to another branch while carrying
the local changes forward when a path was deleted from the index.
* jn/unpack-trees-checkout-m-carry-deletion:
checkout -m: attempt merge when deletion of path was staged
unpack-trees: use 'cuddled' style for if-else cascade
unpack-trees: simplify 'all other failures' case
Teach a few codepaths to punt (instead of dying) when large blobs
that would not fit in core are involved in the operation.
* nd/large-blobs:
diff: shortcut for diff'ing two binary SHA-1 objects
diff --stat: mark any file larger than core.bigfilethreshold binary
diff.c: allow to pass more flags to diff_populate_filespec
sha1_file.c: do not die failing to malloc in unpack_compressed_entry
wrapper.c: introduce gentle xmallocz that does not die()
Add a few more places in "commit" and "checkout" that make sure
that the cache-tree is fully populated in the index.
* dt/cache-tree-repair:
cache-tree: do not try to use an invalidated subtree info to build a tree
cache-tree: Write updated cache-tree after commit
cache-tree: subdirectory tests
test-dump-cache-tree: invalid trees are not errors
cache-tree: create/update cache-tree on checkout
The second batch of the transactional ref update series.
* rs/ref-transaction-1: (22 commits)
update-ref --stdin: pass transaction around explicitly
update-ref --stdin: narrow scope of err strbuf
refs.c: make delete_ref use a transaction
refs.c: make prune_ref use a transaction to delete the ref
refs.c: remove lock_ref_sha1
refs.c: remove the update_ref_write function
refs.c: remove the update_ref_lock function
refs.c: make lock_ref_sha1 static
walker.c: use ref transaction for ref updates
fast-import.c: use a ref transaction when dumping tags
receive-pack.c: use a reference transaction for updating the refs
refs.c: change update_ref to use a transaction
branch.c: use ref transaction for all ref updates
fast-import.c: change update_branch to use ref transactions
sequencer.c: use ref transactions for all ref updates
commit.c: use ref transactions for updates
replace.c: use the ref transaction functions for updates
tag.c: use ref transactions when doing updates
refs.c: add transaction.status and track OPEN/CLOSED
refs.c: make ref_transaction_begin take an err argument
...