This teaches "git-format-patch" to honor the --max-count
parameter revision traversal machinery takes, so that you can
say "git-format-patch -3" to process the three topmost commits
from the current HEAD (or "git-format-patch -2 topic" to name a
specific branch).
Way back when Junio developed the 64 bit index topic he came up
with a means of changing the .idx file format so that older Git
clients would recognize that they don't understand the file and
refuse to read it, while newer clients could tell the difference
between the old-style and new-style .idx files. Unfortunately
this wasn't recorded anywhere.
This change documents how we might go about changing the .idx
file format by using a special signature in the first four bytes.
Credit (and possible blame) goes completely to Junio for thinking
up this technique.
The change also modifies the error message of the current Git code
so that users get a recommendation to upgrade their Git software
should this version or later encounter a new-style .idx which it
cannot process. We already do this for the .pack files, but since
we usually process the .idx files first its important that these
files are recognized and encourage an upgrade.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Refer users to git-rev-parse for revision specification syntax.
The revision specification syntax (sometimes referred to as
SHA1-expressions) is accepted almost everywhere in Git by
almost every tool. Unfortunately it is only documented in
git-rev-parse.txt, and most users don't know to look there.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
In ab2a1a32 Junio improved the reflog query logic to support
obtaining the n-th prior value of a ref, but this was never
documented in git-rev-parse. Now it is.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Documentation: suggest corresponding Porcelain-level in plumbing docs.
Instead of keeping the confused end user reading low-level
documentation, suggest the higher level commands that implement
what the user may want to do using them upfront.
Current README content is way too esoteric for someone looking at GIT
for the first time. Instead it should provide a quick summary of what
GIT is with a few pointers to other resources.
The bulk of the previous README content is moved to
Documentation/core-intro.txt.
Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Branches are only contained by a packfile if the branch actually
had its most recent commit in that packfile. So new branches are
set to MAX_PACK_ID to ensure they don't cause their commit to list
as part of the first packfile when it closes out if the commit was
actually in existance before fast-import started.
Also corrected the type of last_commit to be umaxint_t to prevent
overflow and wraparound on very large imports. Though that is
highly unlikely to occur as we're talking 4 billion commits, which
no real project has right now.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Declare no-arg functions as (void) in fast-import.
Apparently the git convention is to declare any function which
takes no arguments as taking void. I did not do this during the
early fast-import development, but should have.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Correct a few types to be unsigned in fast-import.
The length of an atom string cannot be negative. So make it
explicit and declare it as an unsigned value.
The shift width in a mark table node also cannot be negative.
I'm also moving it to after the pointer arrays to prevent any
possible alignment problems on a 64 bit system.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Corrected BNF input documentation for fast-import.
Now that fast-import uses uintmax_t (the largest available unsigned
integer type) for marks we don't want to say its an unsigned 32
bit integer in ASCII base 10 notation. It could be much larger,
especially on 64 bit systems, and especially if a frontend uses
a very large number of marks (1 per file revision on a very, very
large import).
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
tutorial: shorthand for remotes but show distributed nature of git
* Promiscous pull shows the distributed nature of git better.
* Add a new step after that to teach "remote add".
* Highlight that with the shorthand defined you will get
remote tracking branches for free.
* Fix Alice's workflow.
Signed-off-by: Santi Béjar <sbejar@gmail.com> Signed-off-by: Junio C Hamano <junkio@cox.net>
Turns out that the function WriteMakefile() called in Makefile.PL
outputs the message "Writing perl.mak for Git" to stdout! Thus,
the output of "make -C perl -s --no-print-directory instlibdir"
would be prefixed by that message whenever Makefile.PL was newer
than perl.mak.
This is fixed by redirecting stdout to stderr in Makefile.PL.
Signed-off-by: Johannes E. Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <junkio@cox.net>
Print out the edge commits for each packfile in fast-import.
To help callers repack very large repositories into a series of
packfiles fast-import now outputs the last commits/tags it wrote to
a packfile when it prints out the packfile name. This information
can be feed to pack-objects --revs to repack. For the first pack
of an initial import this is pretty easy (just feed those SHA1s on
stdin) but for subsequent packs you want to feed the subsequent
pack's final SHA1s but also all prior pack's SHA1s prefixed with
the negation operator. This way the prior pack's data does not
get included into the subsequent pack.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Correct object_count type and stat output in fast-import.
Since object_count is limited to 'unsigned long' (really an
unsigned 32 bit integer value) by the pack file format we may as
well use exactly that type here in fast-import for that counter.
An earlier change by me incorrectly made it uintmax_t.
But since object_count is a counter for the current packfile only,
we don't want to output its value at the end. Instead we should
sum up the individual type counters and report that total, as that
will cover all of the packfiles.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
git-svn: print and flush authentication prompts to STDERR
People that redirect STDOUT output should always see STDERR
prompts interactively.
STDERR should always be flushed without buffering, so
they should always show up. If that is unset, we still
explicitly flush by calling STDERR->flush.
The svn command-line client prompts to STDERR, too.
Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <junkio@cox.net>
Much like the pack_sha1 the pack_fd is an unnecessary global
variable, we already have the fd stored in our struct packed_git
*pack_data so that the core library functions in sha1_file.c are
able to lookup and decompress object data that we have previously
written. Keeping an extra copy of this value in our own variable
is just a hold-over from earlier versions of fast-import and is
now completely unnecessary.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Ensure we close the packfile after creating it in fast-import.
Because we are renaming the packfile into its file destination we
need to be sure its not open when the rename is called, otherwise
some operating systems (e.g. Windows) may prevent the rename from
occurring.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Because fast-import automatically updates all references (heads
and tags) at the end of its run the repository is corrupt unless
the objects are available in the .git/objects/pack directory prior
to the refs being modified. The easiest way to ensure that is true
is to move the packfile and its associated index directly into the
.git/objects/pack directory as soon as we have finished output to it.
But the only safe way to do this is to create the a temporary .keep
file for that pack, so we use the same tricks that index-pack uses
when its being invoked by receive-pack.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Rather than maintaing our own packfile level sha1 variable we
can make use of the one already available in struct packed_git.
Its meant for the SHA1 of the index but it can also hold the
SHA1 of the packfile itself between final checksumming of the
packfile and creation of the index.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Replace redundant yread() with read_in_full() in fast-import.
Prior to git having read_in_full() fast-import used its own private
function yread to perform the header reading task. No sense in
keeping that around now that read_in_full is a public, stable
function.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
If a frontend wants to use a mark per file revision and per commit
and is doing a truly huge import (such as a 32 GiB SVN repository)
we may need more than 2**32 unique mark values, especially if the
frontend is unable (or unwilling) to recycle mark values. For mark
idnums we should use the largest unsigned integer type available,
hoping that will be at least 64 bits when we are compiled as a 64
bit executable. This way we may consume huge amounts of memory
storing our mark table, but we'll at least be able to process
the entire import without failing.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Corrected buffer overflow during automatic checkpoint in fast-import.
If we previously were using a delta but we needed to checkpoint the
current packfile and switch to a new packfile we need to throw away
the delta and compress the raw object by itself, as delta chains
cannot span non-thin packfiles. Unfortunately the output buffer
in this case needs to grow, as the size of the compressed object
may be quite a bit larger than the size of the compressed delta.
I've also avoided recompressing the object if we are checkpointing
and we didn't use a delta. In this case the output buffer is the
correct size and has already been populated with the right data,
we just need to close out the current packfile and open a new one.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
git-pull: disallow implicit merging to detached HEAD
Instead, we complain to the user and suggest that they explicitly
specify the remote and branch. We depend on the exit status of
git-symbolic-ref, so let's go ahead and document that.
Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <junkio@cox.net>
Fix git-fetch while on detached HEAD not to give needlessly alarming errors
When we are on a detached HEAD, there is no current branch.
There is no reason to leak the error messages to the end user
since this is a situation we expect to see.
This adds -q option to git-symbolic-ref to exit without issuing
an error message if the given name is not a symbolic ref.
By the way, with or without this patch, there currently is no
good way to tell failure modes between "git symbolic-ref HAED"
and "git symbolic-ref HEAD". Both says "is not a symbolic ref".
Print the packfile names to stdout from fast-import.
Caller scripts may want to know what packfiles the fast-import
process just wrote out for them. This is now output to stdout,
one packfile name per line, after we checkpoint each packfile.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Implemented automatic checkpoints within fast-import.
When the number of objects or number of bytes gets close to the limit
allowed by the packfile format (or configured on the command line by
our caller) we should automatically checkpoint the current packfile
and start a new one before writing the object out. This does however
require that we abandon the delta (if we had one) as its not valid
in a new packfile.
I also added the simple rule that if we got a delta back but the
delta itself is the same size as or larger than the uncompressed
object to ignore the delta and just store the object data. This
should avoid some really bad behavior caused by our current delta
strategy.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Optimize index creation on large object sets in fast-import.
When we are generating multiple packfiles at once we only need
to scan the blocks of object_entry structs which contain objects
for the current packfile. Because the most recent blocks are at
the front of the linked list, and because all new objects going
into the current file are allocated from the front of that list,
we can stop scanning for objects as soon as we identify one which
doesn't belong to the current packfile.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Don't create a final empty packfile in fast-import.
If the last packfile is going to be empty (has 0 objects) then it
shouldn't be kept after the import has terminated, as there is no
point to the packfile. So rather than hashing it and making the
index file, just delete the packfile.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Implemented manual packfile switching in fast-import.
To help importers which are dealing with massive amounts of data
fast-import needs to be able to close the packfile it is currently
writing to and open a new packfile for any additional data that
will be received. A new 'checkpoint' command has been introduced
which can be used by the frontend import process to force this
to occur at any time. This may be useful to ensure a very long
running import doesn't lose any work due to unexpected failures.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Remove unnecessary duplicate_count in fast-import.
There is little reason to be keeping a global duplicate_count
value when we also keep it per object type. The global counter can
easily be computed at the end, once all processing has completed.
This saves us a couple of machine instructions in an unimportant
part of code. But it looks slightly better to me to not keep
two counters around.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Restructure fast-import to support creating multiple packfiles.
Now that we are starting to see some really large projects (such
as KDE or a fork of FreeBSD) get imported into Git we're running
into the upper limit on packfile object count as well as overall
byte length. The KDE and FreeBSD projects are both likely to
require more than 4 GiB to store their current history, which means
we really need multiple packfiles to handle their content.
This is a fairly simple restructuring of the internal code to help
us support creating multiple packfiles from within fast-import.
We are now adding a 5 digit incrementing suffix to the end of the
basename supplied to us by the caller, permitting up to 99,999
packs to be generated in a single fast-import run.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
* git://git.kernel.org/pub/scm/gitk/gitk:
[PATCH] Make gitk work when launched in a subdirectory
[PATCH] gitk: add current directory to main window title
Use nice names in conflict markers during cherry-pick/revert.
Always call the current HEAD 'HEAD', and name the patch being
cherry-picked or reverted by its oneline subject rather than
its SHA1. This matches git am's behavior and is done because
users most commonly are cherry-picking by SHA1 rather than by
ref name.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
This makes revert and cherry-pick to use merge-recursive, to
allow them to notice renames. A pair of test scripts
demonstrate that an old change before a rename happened can be
applied (reverted) after a rename with cherry-pick (with revert).
Documentation: merge-output is not too verbose now.
We've squelched output from merge-recursive, and git-merge when
used with recursive does not attempt the trivial one first
anymore, so there won't be "Trying ... Nope." messages now.
Remove hash in git-describe in favor of util slot.
Currently we don't use the util field of struct commit but we want
fast access to the highest priority name that references any given
commit object during our matching loop. A really simple approach
is to just store the name directly in the util field.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Correct priority of lightweight tags in git-describe.
We really want to always favor an annotated tag over a lightweight
tag when describing a commit. Unfortunately git-describe wasn't
doing this as it was favoring the depth attribute of a possible_tag
over the priority. Now priority is the highest sort and we only
consider a lightweight tag if no annotated tags were identified.
Rather than searching for the minimum tag using a simple loop we
now sort them using a stable sort algorithm, this way the possible
tags display in order if --debug gets used. The stable sort helps
to preseve the inherit topology/date order that we obtain during
our search loop.
This fix allows the tests in t6120-describe.sh to pass.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Improve git-describe performance by reducing revision listing.
My prior version of git-describe ran very slowly on even reasonably
sized projects like git.git and linux.git as it tended to identify
a large number of possible tags and then needed to generate the
revision list for each of those tags to sort them and select the
best tag to describe the input commit.
All we really need is the number of commits in the input revision
which are not in the tag. We can generate these counts during
the revision walking and tag matching loop by assigning a color to
each tag and coloring the commits as we walk them. This limits us
to identifying no more than 26 possible tags, as there is limited
space available within the flags field of struct commit.
The limitation of 26 possible tags is hopefully not going to be a
problem in real usage, as most projects won't create 26 maintenance
releases and merge them back into a development trunk after the
development trunk was tagged with a release candidate tag. If that
does occur git-describe will start to revert to its old behavior of
using the newer maintenance release tag to describe the development
trunk, rather than the development trunk's own tag. The suggested
workaround would be to retag the development trunk's tip.
However since even 26 possible tags can take a while to generate a
description for on some projects I'm defaulting the limit to 10 but
offering the user --candidates to increase the number of possible
matches if they need a more accurate result. I specifically chose
10 for the default as it seems unlikely projects will have more
than 10 maintenance releases merged into a development trunk before
retagging the development trunk, and it seems to perform about the
same on linux.git as v1.4.4.4 git-describe.
A large amount of debugging information was also added during
the development of this change, so I've left it in to be toggled
on with --debug. It may be useful to the end user to help them
understand why git-describe took one particular tag over another.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Use binary searching on large buckets in git-describe.
If a project has a really huge number of tags (such as several
thousand tags) then we are likely to have nearly a hundred tags in
some buckets. Scanning those buckets as linked lists could take
a large amount of time if done repeatedly during history traversal.
Since we are searching for a unique commit SHA1 we can sort all
tags by commit SHA1 and perform a binary search within the bucket.
Once we identify a particular tag as matching this commit we walk
backwards within the bucket matches to make sure we pick up the
highest priority tag for that commit, as the binary search may
have landed us in the middle of a set of tags which point at the
same commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
If a project has a very large number of tags then git-describe
will spend a good part of its time looping over the tags testing
them one at a time to determine if it matches a given commit.
For 10 tags this is not a big deal, but for hundreds of tags the
time could become considerable if we don't find an exact match for
the input commit and we need to walk back along the history chain.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Several people have suggested that its always better to describe
a commit using an annotated tag, and to only use a lightweight tag
if absolutely no annotated tag matches the input commit.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Improve reuse of sha1_file library within fast-import.
Now that the sha1_file.c library routines use the sliding mmap
routines to perform efficient access to portions of a packfile
I can remove that code from fast-import.c and just invoke it.
One benefit is we now have reloading support for any packfile which
uses OBJ_OFS_DELTA. Another is we have significantly less code
to maintain.
This code reuse change *requires* that fast-import generate only
an OBJ_OFS_DELTA format packfile, as there is absolutely no index
available to perform OBJ_REF_DELTA lookup in while unpacking
an object. This is probably reasonable to require as the delta
offsets result in smaller packfiles and are faster to unpack,
as no index searching is required. Its also only a temporary
requirement as users could always repack without offsets before
making the import available to older versions of Git.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Convert output messages in merge-recursive to past tense.
Now that we are showing the output messages for verbosity levels
<5 after all actions have been performed (due to the progress meter
running during the actions) it can be confusing to see messages in
the present tense when the user is looking at a '100% done' message
right above them. Converting the messages to past tense will appear
more correct in this case, and shouldn't affect a developer who is
debugging the application and running it at a verbosity level >=5.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Because large merges on slow systems can take up to a minute to
execute we should try to keep the user entertained with a progress
meter to let them know how far we have progressed through the
current merge.
The progress meter considers each entry in the in-memory index to
be a unit, which means a single recursive merge will double the
number of units in the progress meter. Files which are unmerged
after the 3-way tree merge are also considered a unit within the
progress meter.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Buffering all message output until a merge invocation is complete is
necessary to prevent intereferring with a progress meter that would
indicate the number of files completely merged, and how many remain.
This change does not introduce a progress meter, but merely lays
the groundwork to buffer the output.
To aid debugging output buffering is only enabled if verbosity
is lower than 5. When using verbosity levels above 5 the user is
probably debugging the merge program itself and does not want to
see the output delayed, especially if they are stepping through
portions of the code in a debugger.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Allow the user to control the verbosity of merge-recursive.
Junio C Hamano <junkio@cox.net> writes:
>
> I think the output from merge-recursive can be categorized into 5
> verbosity levels:
>
> 1. "CONFLICT", "Rename", "Adding here instead due to D/F conflict"
> (outermost)
>
> 2. "Auto-merged successfully" (outermost)
>
> 3. The first "Merging X with Y".
>
> 4. outermost "Merging:\ntitle1\ntitle2".
>
> 5. outermost "found N common ancestors\nancestor1\nancestor2\n..."
> and anything from inner merge.
>
> I would prefer the default verbosity level to be 2 (that is, show
> both 1 and 2).
and this change makes it so. I think level 3 is probably pointless
as its only one line of output above level 2, but I can see how some
users may want to view it but not view the slightly more verbose
output of level 4.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Remove unnecessary call_depth parameter in merge-recursive.
Because the output_indent always matches the call_depth value
there is no reason to pass around the call_depth to the merge
function during each recursive invocation.
This is a simple refactoring that will make the code easier to
follow later on as I start to add output verbosity controls.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/int:
More tests in t3901.
Consistent message encoding while reusing log from an existing commit.
t3901: test "format-patch | am" pipe with i18n
Use log output encoding in --pretty=email headers.
* jc/subdir:
Allow whole-tree operations to be started from a subdirectory
Use cd_to_toplevel in scripts that implement it by hand.
Define cd_to_toplevel shell function in git-sh-setup
Remove read_or_die in favor of better error messages.
Originally I introduced read_or_die for the purpose of reading
the pack header and trailer, and I was too lazy to print proper
error messages.
Linus Torvalds <torvalds@osdl.org>:
> For a read error, at the very least you have to say WHICH FILE
> couldn't be read, because it's usually a matter of some file just
> being too short, not some system-wide problem.
and of course Linus is right. Make it so.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
Hide output about SVN::Core not being found during tests.
If the user doesn't have SVN::Core installed or working then the
SVN tests properly turn themselves off. But the user doesn't need
to know that SVN::Core isn't loadable as a Perl module. Unless of
course they are trying to debug the test, so lets relegate the Perl
failures to --verbose only.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
I'm bringing master in early so that the OBJ_OFS_DELTA implementation
is available as part of the topic. This way git-fast-import can
learn about this new slightly smaller and faster packfile format,
and can generate them directly rather than needing to have them be
repacked with git-pack-objects.
Due to the API changes in master during the period of development
of git-fast-import, a few minor tweaks to fast-import.c are needed
to produce a working merge. I've done them here as part of the
merge to ensure bisection always works.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Allow creating branches without committing in fast-import.
Some importers may want to create a branch long before they actually
commit to it, or in some cases they may never commit to the branch
but they still need the ref to be created in the repository after
the import is complete.
This extends the 'reset ' command to automatically create a new
branch if the supplied reference isn't already known as a branch.
While I'm at it I also modified the syntax of the reset command
to terminate with an empty line, like commit and tag operate.
This just makes the command set more consistent.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Some importers are able to determine when branch merges occurred
within their source data. In these cases they will want to supply
the correct commits to fast-import so that a proper merge commit
will exist in Git. This is now supported by supplying a 'merge '
command after the commit message and optional from command.
A merge is not actually performed by fast-import, its assumed that
the frontend performed any sort of merging activity already and
that fast-import should simply be storing its result.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Fix repository corruption when using marks for modified blobs.
Apparently we did not copy the blob SHA1 into the stack variable
'sha1' when a mark is used to refer to a prior blob. This code
was not previously tested as the Mozilla CVS -> git-fast-import
program always fed us full SHA1s for modified blobs and did not
use the mark feature there.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The new tree delta implementation caused blob SHA1s to be used
instead of a tree SHA1 when a tree was written out. This really
only appeared to happen when converting an existing file to a tree,
but may have been possible in some other situations.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>