cat-file: disable object/refname ambiguity check for batch mode
A common use of "cat-file --batch-check" is to feed a list
of objects from "rev-list --objects" or a similar command.
In this instance, all of our input objects are 40-byte sha1
ids. However, cat-file has always allowed arbitrary revision
specifiers, and feeds the result to get_sha1().
Fortunately, get_sha1() recognizes a 40-byte sha1 before
doing any hard work trying to look up refs, meaning this
scenario should end up spending very little time converting
the input into an object sha1. However, since 798c35f
(get_sha1: warn about full or short object names that look
like refs, 2013-05-29), when we encounter this case, we
spend the extra effort to do a refname lookup anyway, just
to print a warning. This is further exacerbated by ca91993
(get_packed_ref_cache: reload packed-refs file when it
changes, 2013-06-20), which makes individual ref lookup more
expensive by requiring a stat() of the packed-refs file for
each missing ref.
With no patches, this is the time it takes to run:
If we revert ca91993, the packed-refs up-to-date check, it
gets a little better:
real 0m54.697s
user 0m21.692s
sys 0m32.916s
but we are still spending quite a bit of time on ref lookup
(and we would not want to revert that patch, anyway, which
has correctness issues). If we revert 798c35f, disabling
the warning entirely, we get a much more reasonable time:
real 0m7.452s
user 0m6.836s
sys 0m0.608s
This patch does the moral equivalent of this final case (and
gets similar speedups). We introduce a global flag that
callers of get_sha1() can use to avoid paying the price for
the warning.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The pack revindex stores the offsets of the objects in the
pack in sorted order, allowing us to easily find the on-disk
size of each object. To compute it, we populate an array
with the offsets from the sha1-sorted idx file, and then use
qsort to order it by offsets.
That does O(n log n) offset comparisons, and profiling shows
that we spend most of our time in cmp_offset. However, since
we are sorting on a simple off_t, we can use numeric sorts
that perform better. A radix sort can run in O(k*n), where k
is the number of "digits" in our number. For a 64-bit off_t,
using 16-bit "digits" gives us k=4.
On the linux.git repo, with about 3M objects to sort, this
yields a 400% speedup. Here are the best-of-five numbers for
running
echo HEAD | git cat-file --batch-check="%(objectsize:disk)
on a fully packed repository, which is dominated by time
spent building the pack revindex:
before after
real 0m0.834s 0m0.204s
user 0m0.788s 0m0.164s
sys 0m0.040s 0m0.036s
This matches our algorithmic expectations. log(3M) is ~21.5,
so a traditional sort is ~21.5n. Our radix sort runs in k*n,
where k is the number of radix digits. In the worst case,
this is k=4 for a 64-bit off_t, but we can quit early when
the largest value to be sorted is smaller. For any
repository under 4G, k=2. Our algorithm makes two passes
over the list per radix digit, so we end up with 4n. That
should yield ~5.3x speedup. We see 4x here; the difference
is probably due to the extra bucket book-keeping the radix
sort has to do.
On a smaller repo, the difference is less impressive, as
log(n) is smaller. For git.git, with 173K objects (but still
k=2), we see a 2.7x improvement:
before after
real 0m0.046s 0m0.017s
user 0m0.036s 0m0.012s
sys 0m0.008s 0m0.000s
On even tinier repos (e.g., a few hundred objects), the
speedup goes away entirely, as the small advantage of the
radix sort gets erased by the book-keeping costs (and at
those sizes, the cost to generate the the rev-index gets
lost in the noise anyway).
Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
pack-revindex: use unsigned to store number of objects
A packfile may have up to 2^32-1 objects in it, so the
"right" data type to use is uint32_t. We currently use a
signed int, which means that we may behave incorrectly for
packfiles with more than 2^31-1 objects on 32-bit systems.
Nobody has noticed because having 2^31 objects is pretty
insane. The linux.git repo has on the order of 2^22 objects,
which is hundreds of times smaller than necessary to trigger
the bug.
Let's bump this up to an "unsigned". On 32-bit systems, this
gives us the correct data-type, and on 64-bit systems, it is
probably more efficient to use the native "unsigned" than a
true uint32_t.
While we're at it, we can fix the binary search not to
overflow in such a case if our unsigned is 32 bits.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If we get an input line to --batch or --batch-check that
looks like "HEAD foo bar", we will currently feed the whole
thing to get_sha1(). This means that to use --batch-check
with `rev-list --objects`, one must pre-process the input,
like:
Besides being more typing and slightly less efficient to
invoke `cut`, the result loses information: we no longer
know which path each object was found at.
This patch teaches cat-file to split input lines at the
first whitespace. Everything to the left of the whitespace
is considered an object name, and everything to the right is
made available as the %(reset) atom. So you can now do:
git rev-list --objects HEAD |
git cat-file --batch-check='%(objectsize) %(rest)'
to collect object sizes at particular paths.
Even if %(rest) is not used, we always do the whitespace
split (which means you can simply eliminate the `cut`
command from the first example above).
This whitespace split is backwards compatible for any
reasonable input. Object names cannot contain spaces, so any
input with spaces would have resulted in a "missing" line.
The only input hurt is if somebody really expected input of
the form "HEAD is a fine-looking ref!" to fail; it will now
parse HEAD, and make "is a fine-looking ref!" available as
%(rest).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This atom is just like %(objectsize), except that it shows
the on-disk size of the object rather than the object's true
size. In other words, it makes the "disk_size" query of
sha1_object_info_extended available via the command-line.
This can be used for rough attribution of disk usage to
particular refs, though see the caveats in the
documentation.
This patch does not include any tests, as the exact numbers
returned are volatile and subject to zlib and packing
decisions. We cannot even reliably guarantee that the
on-disk size is smaller than the object content (though in
general this should be the case for non-trivial objects).
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `cat-file --batch-check` command can be used to quickly
get information about a large number of objects. However, it
provides a fixed set of information.
This patch adds an optional <format> option to --batch-check
to allow a caller to specify which items they are interested
in, and in which order to output them. This is not very
exciting for now, since we provide the same limited set that
you could already get. However, it opens the door to adding
new format items in the future without breaking backwards
compatibility (or forcing callers to pay the cost to
calculate uninteresting items).
Since the --batch option shares code with --batch-check, it
receives the same feature, though it is less likely to be of
interest there.
The format atom names are chosen to match their counterparts
in for-each-ref. Though we do not (yet) share any code with
for-each-ref's formatter, this keeps the interface as
consistent as possible, and may help later on if the
implementations are unified.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
We currently use an int to tell us whether --batch parsing
is on, and if so, whether we should print the full object
contents. Let's instead factor this into a struct, filled in
by callback, which will make further batch-related options
easy to add.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The regular "git cat-file -p" and "git cat-file blob" code
paths already learned to stream large blobs. Let's do the
same here.
Note that this means we look up the type and size before
making a decision of whether to load the object into memory
or stream (just like the "-p" code path does). That can lead
to extra work, but it should be dwarfed by the cost of
actually accessing the object itself. In my measurements,
there was a 1-2% slowdown when using "--batch" on a large
number of objects.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In modern tests, we typically put output into a file and
compare it with test_cmp. This is nicer than just comparing
via "test", and much shorter than comparing via "test" and
printing a custom message.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
teach sha1_object_info_extended a "disk_size" query
Using sha1_object_info_extended, a caller can find out the
type of an object, its size, and information about where it
is stored. In addition to the object's "true" size, it can
also be useful to know the size that the object takes on
disk (e.g., to generate statistics about which refs consume
space).
This patch adds a "disk_sizep" field to "struct object_info",
and fills it in during sha1_object_info_extended if it is
non-NULL.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The sha1_object_info_extended function expects the caller to
provide a "struct object_info" which contains pointers to
"query" items that will be filled in. The purpose of
providing pointers rather than storing the response directly
in the struct is so that callers can choose not to incur the
expense in finding particular fields that they do not care
about.
Right now the only query item is "sizep", and all callers
set it explicitly to choose whether or not to query it; they
can then leave the rest of the struct uninitialized.
However, as we add new query items, each caller will have to
be updated to explicitly turn off the new ones (by setting
them to NULL). Instead, let's teach each caller to
zero-initialize the struct, so that they do not have to
learn about each new query item added.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
get_sha1: warn about full or short object names that look like refs
When we get 40 hex digits, we immediately assume it's an SHA-1. This
is the right thing to do because we have no way else to specify an
object. If there is a ref with the same object name, it will be
ignored. Warn the user about this case because the ref with full
object name is likely a mistake, for example
zsh completion wrapper doesn't reimplement __gitcompadd(). Although it
should be trivial to do that, let's use __gitcomp_nl() which achieves
exactly the same thing, specially since the suffix ($4) has to be empty.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* git://git.bogomips.org/git-svn:
git-svn: introduce --parents parameter for commands branch and tag
git-svn: clarify explanation of --destination argument
git-svn: multiple fetch/branches/tags keys are supported
git-svn: clarify explanation of --destination argument
The existing documentation for "-d" does not make it obvious whether
its argument is supposed to be a full svn path, a partial svn path,
the glob from the config file, or what. Clarify the text and add an
example to get the reader started.
Reported-by: Nathan Gray <n8gray@n8gray.org> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
git-svn: multiple fetch/branches/tags keys are supported
"git svn" can be configured to use multiple fetch, branches, and tags
refspecs by passing multiple --branches or --tags options at init time
or editing the configuration file later, which can be handy when
working with messy Subversion repositories. Add a note to the
configuration section documenting how this works.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
git clone hangs on windows, and file.write would return errno 22 inside
of mercurial's windows.winstdout wrapper class. This patch sets stdout's
mode to binary, fixing both issues.
[fc: cleaned up]
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
difftool: fix dir-diff when file does not exist in working tree
Commit 02c5631 (difftool --dir-diff: symlink all files matching the
working tree, 2013-03-14) does not handle the case where a file that is
being compared does not exist in the working tree. Fix this by checking
for existence explicitly before running git-hash-object.
Reported-by: Kevin Bracey <kevin@bracey.fi> Signed-off-by: John Keeping <john@keeping.me.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Different versions of Mercurial have different arguments for
bookmarks.updatefromremote(), while it should be possible to call the
right function with the right arguments depending on the version, it's
safer to restore the old behavior for now.
Reported by Rodney Lorrimar.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-submodule.txt: Clarify 'init' and 'add' subcommands.
Describe how 'add' sets the submodule's logical name, which is used in
the configuration entry names.
Clarify that 'init' only sets up the configuration entries for
submodules that have already been added elsewhere. Describe that
<path> arguments limit the submodules that are configured.
Signed-off-by: Dale Worley <worley@ariadne.com> Acked-by: Jens Lehmann <Jens.Lehmann@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 95b0c60 (remote-bzr: add support for bzr repos) introduced a
regression by assuming all bzr remote repos are listable, but they are
not.
If they are not listable they are basically useless, so let's assume
there is no bzr repo.
Reported-by: Thorsten Kranzkowski <dl8bcu@dl8bcu.de> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
In certain situations we might end up pushing garbage revisions
(e.g. in a rebase), and the patches to deal with that haven't been
merged yet. So let's disable forced pushes by default.
We are essentially reverting back to the old v1.8.2 behavior, to
minimize the possibility of regressions, but in a way the user can
configure.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If a clone exists with the old organization (v1.8.2) it will prevent
the new shared bzr repository organization from working, so let's
remove this repository, which is not used any more.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* git://ozlabs.org/~paulus/gitk:
gitk: On OSX, bring the gitk window to front
gitk: Add support for -G'regex' pickaxe variant
gitk: Add menu item for reverting commits
gitk: Simplify file filtering
gitk: Display the date of a tag in a human-friendly way
gitk: Improve behaviour of drop-down lists
gitk: Move hard-coded colors to .gitk
On OSX, Tcl/Tk application windows are created behind all
the applications down the stack of windows. This is very
annoying, because once a gitk window appears, it's the
downmost window and switching to it is pain.
The patch is: if we are on OSX, use osascript to
bring the current Wish process window to front.
Signed-off-by: Tair Sabirgaliev <tair.sabirgaliev@gmail.com>
Thanks-to: Stefan Haller <lists@haller-berlin.de> Signed-off-by: Paul Mackerras <paulus@samba.org>
gitk: Display the date of a tag in a human-friendly way
By selecting a tag within gitk you can display information about it.
This information is output by using the command
'git cat-file tag <tagid>'
This outputs the *raw* information from the tag, amongst which is the
time - in seconds since the epoch. As useful as that value is, I find it
a lot easier to read and process time which it is something like:
"Mon Dec 31 14:26:11 2012 -0800"
This change will modify the display of tags in gitk like so:
@@ -1,7 +1,7 @@
object 5d417842efeafb6e109db7574196901c4e95d273
type commit
tag v1.8.1
-tagger Junio C Hamano <gitster@pobox.com> 1356992771 -0800
+tagger Junio C Hamano <gitster@pobox.com> Mon Dec 31 14:26:11 2012 -0800
Git 1.8.1
-----BEGIN PGP SIGNATURE-----
Signed-off-by: Anand Kumria <wildfire@progsoc.org> Signed-off-by: Paul Mackerras <paulus@samba.org>
The drop-down lists used for things like the criteria for finding
commits (containing/touching paths/etc.) use a combobox if we are
using the ttk widgets. By default the combobox exports its value
as the selection when it is changed, which is unnecessary, and sometimes
the combobox wouldn't release the selection, which is annoying.
To fix this, we make these comboboxes not export their selection,
and also clear their selection whenever they are changed. This makes
them more like a simple selection of alternatives, improving the look
and feel of gitk.
CodingGuidelines: Documentation/*.txt are the sources
People not familiar with AsciiDoc may not realize they are
supposed to update *.txt files and not *.html/*.1 files when
preparing patches to the project.
Signed-off-by: Dale Worley <worley@ariadne.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* maint:
Git 1.8.2.3
t5004: avoid using tar for checking emptiness of archive
t5004: ignore pax global header file
mergetools/kdiff3: do not use --auto when diffing
transport-helper: trivial style cleanup
cherry-pick: picking a tag that resolves to a commit is OK
Earlier, 21246dbb9e0a (cherry-pick: make sure all input objects are
commits, 2013-04-11) tried to catch an unlikely "git cherry-pick $blob"
as an error, but broke a more important use case to cherry-pick a
tag that points at a commit.
t5004: avoid using tar for checking emptiness of archive
Test 2 of t5004 checks if a supposedly empty tar archive really
contains no files. 24676f02 (t5004: fix issue with empty archive test
and bsdtar) removed our commit hash to make it work with bsdtar, but
the test still fails on NetBSD and OpenBSD, which use their own tar
that considers a tar file containing only NULs as broken.
Here's what the different archivers do when asked to create a tar
file without entries:
So NetBSD's native tar creates an empty file, while GNU tar and bsdtar
both give us 10KB of NULs -- just like git archive with an empty tree.
Now let's see how the archivers handle these two kinds of empty tar
files:
$ tar tf zero.tar; echo $?
tar: Unexpected EOF on archive file
1
$ gtar tf zero.tar; echo $?
gtar: This does not look like a tar archive
gtar: Exiting with failure status due to previous errors
2
$ bsdtar tf zero.tar; echo $?
0
$ tar tf tenk.tar; echo $?
tar: Cannot identify format. Searching...
tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
1
$ gtar tf tenk.tar; echo $?
0
$ bsdtar tf tenk.tar; echo $?
0
NetBSD's tar complains about both, bsdtar happily accepts any of them
and GNU tar doesn't like zero-length archive files. So the safest
course of action is to stay with our block-of-NULs format which is
compatible with GNU tar and bsdtar, as we can't make NetBSD's native
tar happy anyway.
We can simplify our test, however, by taking tar out of the picture.
Instead of extracting the archive and checking for the non-presence of
files, check if the file has a size of 10KB and contains only NULs.
This makes t5004 pass on NetBSD and OpenBSD.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Versions of tar that don't know pax headers -- like the ones in NetBSD 6
and OpenBSD 5.2 -- extract them as regular files. Explicitly ignore the
file created for our global header when checking the list of extracted
files, as this is normal and harmless fall-back behaviour. This fixes
test 3 of t5004 on these platforms.
Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The `kdiff3 --auto` help message is, "No GUI if all conflicts are auto-
solvable." This flag was carried over from the original mergetool
commands. diff_cmd() is for two-way comparisons only so remove the
superfluous flag.
Signed-off-by: David Aguilar <davvid@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
The SVN::Fetcher module is now able to filter for inclusion as well
as exclusion (as used by --ignore-path). Also added tests, documentation
changes and git completion script.
If you have an SVN repository with many top level directories and you
only want a git-svn clone of some of them then using --ignore-path is
difficult as it requires a very long regexp. In this case it's much
easier to filter for inclusion.
[ew: remove trailing whitespace]
Signed-off-by: Paul Walmsley <pjwhams@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
Git::SVN::*: add missing "NAME" section to perldoc
lexgrog(1) relies on the NAME section to find a manpage's subject's
name and description for easy access later using "man -k". Add the
section it expects.
Noticed using lintian.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Eric Wong <normalperson@yhbt.net>
When svn.pushmergeinfo is set, the target branch is included in the
mergeinfo if it was previously merged into one of the source branches.
SVN does not do this.
Remove merge target branch path from resulting mergeinfo when
svn.pushmergeinfo is set to better match the behavior of SVN. Update the
svn-mergeinfo-push test.
[ew: 80 columns]
Signed-off-by: Michael Contreras <michael@inetric.com> Reported-by: Avishay Lavie <avishay.lavie@gmail.com> Acked-by: Eric Wong <normalperson@yhbt.net>
It's possible that the previous tip goes away, we should not assume it's
always present. Fortunately we are only using it to calculate the
progress to display to the user, so only that needs to be fixed.
Also, add a test that triggers this issue.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
git-merge-tree causes a null pointer dereference when a directory
entry exists in only one or two of the three trees being compared with
no corresponding entry in the other tree(s).
When this happens, we want to handle the entry as a directory and not
attempt to mark it as a file merge. Do this by setting the entries bit
in the directory mask when the entry is missing or when it is a
directory, only performing the file comparison when we know that a file
entry exists.
Reported-by: Andreas Jacobsen <andreas@andreasjacobsen.com> Signed-off-by: John Keeping <john@keeping.me.uk> Tested-by: Andreas Jacobsen <andreas@andreasjacobsen.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* fc/remote-bzr:
remote-bzr: avoid bad refs
remote-bzr: convert all unicode keys to str
remote-bzr: access branches only when needed
remote-bzr: delay peer branch usage
remote-bzr: iterate revisions properly
remote-bzr: improve progress reporting
remote-bzr: add option to specify branches
remote-bzr: add custom method to find branches
remote-bzr: improve author sanitazion
remote-bzr: add support for shared repo
remote-bzr: fix branch names
remote-bzr: add support for bzr repos
remote-bzr: use branch variable when appropriate
remote-bzr: fix partially pushed merge
remote-bzr: fixes for branch diverge
remote-bzr: add support to push merges
remote-bzr: always try to update the worktree
remote-bzr: fix order of locking in CustomTree
remote-bzr: delay blob fetching until the very end
remote-bzr: cleanup CustomTree
Versions of fast-export before v1.8.2 throws a bad 'reset' commands
because of a behavior in transport-helper that is not even needed.
We should ignore them, otherwise we will treat them as branches and
fail.
This was fixed in v1.8.2, but some people use this script in older
versions of git.
Also, check if the ref was a tag, and skip it for now.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
If there's already a remote-helper tracking ref, we can fetch the
SHA-1 to report proper push messages (as opposed to always reporting
[new branch]).
The remote-helper currently can specify the old SHA-1 to avoid this
problem, but there's no point in forcing all remote-helpers to be aware
of git commit ids; they should be able to be agnostic of them.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Reviewed-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Merge branch 'tr/remote-tighten-commandline-parsing' into maint
* tr/remote-tighten-commandline-parsing:
remote: 'show' and 'prune' can take more than one remote
remote: check for superfluous arguments in 'git remote add'
remote: add a test for extra arguments, according to docs
* rs/pp-user-info-without-extra-allocation:
pretty: remove intermediate strbufs from pp_user_info()
pretty: simplify output line length calculation in pp_user_info()
pretty: simplify input line length calculation in pp_user_info()
* tr/remote-tighten-commandline-parsing:
remote: 'show' and 'prune' can take more than one remote
remote: check for superfluous arguments in 'git remote add'
remote: add a test for extra arguments, according to docs
Bazaar doesn't seem to be tested for multiple usage of branches, so
resources seem to be leaked all over. Let's try to minimize this by
accessing the Branch objects only when needed.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This way we don't need to store the list of all the revisions, which
doesn't seem to be very memory efficient with bazaar's design, for
whatever reason.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>