From: Junio C Hamano <gitster@pobox.com>
Date: Mon, 1 Aug 2011 22:00:14 +0000 (-0700)
Subject: Merge branch 'sr/transport-helper-fix'
X-Git-Tag: v1.7.7-rc0~61
X-Git-Url: https://git.lorimer.id.au/gitweb.git/diff_plain/59d9ba869e900bf4da9822f417e4d40a710d8063?ds=inline;hp=-c

Merge branch 'sr/transport-helper-fix'

* sr/transport-helper-fix: (21 commits)
transport-helper: die early on encountering deleted refs
transport-helper: implement marks location as capability
transport-helper: Use capname for refspec capability too
transport-helper: change import semantics
transport-helper: update ref status after push with export
transport-helper: use the new done feature where possible
transport-helper: check status code of finish_command
transport-helper: factor out push_update_refs_status
fast-export: support done feature
fast-import: introduce 'done' command
git-remote-testgit: fix error handling
git-remote-testgit: only push for non-local repositories
remote-curl: accept empty line as terminator
remote-helpers: export GIT_DIR variable to helpers
git_remote_helpers: push all refs during a non-local export
transport-helper: don't feed bogus refs to export push
git-remote-testgit: import non-HEAD refs
t5800: document some non-functional parts of remote helpers
t5800: use skip_all instead of prereq
t5800: factor out some ref tests
...
---

59d9ba869e900bf4da9822f417e4d40a710d8063
diff --combined Documentation/git-fast-export.txt
index a29ac021d9,e3f845371a..f37eada63a
--- a/Documentation/git-fast-export.txt
+++ b/Documentation/git-fast-export.txt
@@@ -8,7 -8,6 +8,7 @@@ git-fast-export - Git data exporte
  
  SYNOPSIS
  --------
 +[verse]
  'git fast-export [options]' | 'git fast-import'
  
  DESCRIPTION
@@@ -83,6 -82,10 +83,10 @@@ marks the same across runs
  	allow that.  So fake a tagger to be able to fast-import the
  	output.
  
+ --use-done-feature::
+ 	Start the stream with a 'feature done' stanza, and terminate
+ 	it with a 'done' command.
+ 
  --no-data::
  	Skip output of blob objects and instead refer to blobs via
  	their original SHA-1 hash.  This is useful when rewriting the
diff --combined Documentation/git-fast-import.txt
index 95e480ef79,0fc68a911e..2969388880
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@@ -8,7 -8,6 +8,7 @@@ git-fast-import - Backend for fast Git 
  
  SYNOPSIS
  --------
 +[verse]
  frontend | 'git fast-import' [options]
  
  DESCRIPTION
@@@ -102,6 -101,12 +102,12 @@@ OPTION
  	when the `cat-blob` command is encountered in the stream.
  	The default behaviour is to write to `stdout`.
  
+ --done::
+ 	Require a `done` command at the end of the stream.
+ 	This option might be useful for detecting errors that
+ 	cause the frontend to terminate before it has started to
+ 	write a stream.
+ 
  --export-pack-edges=<file>::
  	After creating a packfile, print a line of data to
  	<file> listing the filename of the packfile and the last
@@@ -331,6 -336,11 +337,11 @@@ and control the current import process
  	standard output.  This command is optional and is not needed
  	to perform an import.
  
+ `done`::
+ 	Marks the end of the stream. This command is optional
+ 	unless the `done` feature was requested using the
+ 	`--done` command line option or `feature done` command.
+ 
  `cat-blob`::
  	Causes fast-import to print a blob in 'cat-file --batch'
  	format to the file descriptor set with `--cat-blob-fd` or
@@@ -649,14 -659,9 +660,14 @@@ paths for a commit are encouraged to d
  
  `notemodify`
  ^^^^^^^^^^^^
 -Included in a `commit` command to add a new note (annotating a given
 -commit) or change the content of an existing note.  This command has
 -two different means of specifying the content of the note.
 +Included in a `commit` `<notes_ref>` command to add a new note
 +annotating a `<committish>` or change this annotation contents.
 +Internally it is similar to filemodify 100644 on `<committish>`
 +path (maybe split into subdirectories). It's not advised to
 +use any other commands to write to the `<notes_ref>` tree except
 +`filedeleteall` to delete all existing notes in this tree.
 +This command has two different means of specifying the content
 +of the note.
  
  External data format::
  	The data content for the note was already supplied by a prior
@@@ -1021,6 -1026,11 +1032,11 @@@ notes:
  	Versions of fast-import not supporting notes will exit
  	with a message indicating so.
  
+ done::
+ 	Error out if the stream ends without a 'done' command.
+ 	Without this feature, errors causing the frontend to end
+ 	abruptly at a convenient point in the stream can go
+ 	undetected.
  
  `option`
  ~~~~~~~~
@@@ -1050,6 -1060,15 +1066,15 @@@ not be passed as option
  * cat-blob-fd
  * force
  
+ `done`
+ ~~~~~~
+ If the `done` feature is not in use, treated as if EOF was read.
+ This can be used to tell fast-import to finish early.
+ 
+ If the `--done` command line option or `feature done` command is
+ in use, the `done` command is mandatory and marks the end of the
+ stream.
+ 
  Crash Reports
  -------------
  If fast-import is supplied invalid input it will terminate with a
diff --combined Documentation/git-remote-helpers.txt
index 930b4034ac,18b8341f24..4f83dea5a3
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@@ -7,7 -7,6 +7,7 @@@ git-remote-helpers - Helper programs t
  
  SYNOPSIS
  --------
 +[verse]
  'git remote-<transport>' <repository> [<URL>]
  
  DESCRIPTION
@@@ -48,6 -47,9 +48,9 @@@ arguments. The first argument specifie
  it is either the name of a configured remote or a URL. The second
  argument specifies a URL; it is usually of the form
  '<transport>://<address>', but any arbitrary string is possible.
+ The 'GIT_DIR' environment variable is set up for the remote helper
+ and can be used to determine where to store additional data or from
+ which directory to invoke auxiliary git commands.
  
  When git encounters a URL of the form '<transport>://<address>', where
  '<transport>' is a protocol that it cannot handle natively, it
diff --combined fast-import.c
index 9e8d1868aa,8a8a915897..7cc22625e5
--- a/fast-import.c
+++ b/fast-import.c
@@@ -304,7 -304,6 +304,7 @@@ static unsigned int atom_cnt
  static struct atom_str **atom_table;
  
  /* The .pack file being generated */
 +static struct pack_idx_option pack_idx_opts;
  static unsigned int pack_id;
  static struct sha1file *pack_file;
  static struct packed_git *pack_data;
@@@ -355,6 -354,7 +355,7 @@@ static unsigned int cmd_save = 100
  static uintmax_t next_mark;
  static struct strbuf new_data = STRBUF_INIT;
  static int seen_data_command;
+ static int require_explicit_termination;
  
  /* Signal handling */
  static volatile sig_atomic_t checkpoint_requested;
@@@ -897,7 -897,7 +898,7 @@@ static const char *create_index(void
  	if (c != last)
  		die("internal consistency error creating the index");
  
 -	tmpfile = write_idx_file(NULL, idx, object_count, pack_data->sha1);
 +	tmpfile = write_idx_file(NULL, idx, object_count, &pack_idx_opts, pack_data->sha1);
  	free(idx);
  	return tmpfile;
  }
@@@ -1018,7 -1018,7 +1019,7 @@@ static int store_object
  	unsigned char sha1[20];
  	unsigned long hdrlen, deltalen;
  	git_SHA_CTX c;
 -	z_stream s;
 +	git_zstream s;
  
  	hdrlen = sprintf((char *)hdr,"%s %lu", typename(type),
  		(unsigned long)dat->len) + 1;
@@@ -1051,7 -1051,7 +1052,7 @@@
  		delta = NULL;
  
  	memset(&s, 0, sizeof(s));
 -	deflateInit(&s, pack_compression_level);
 +	git_deflate_init(&s, pack_compression_level);
  	if (delta) {
  		s.next_in = delta;
  		s.avail_in = deltalen;
@@@ -1059,11 -1059,11 +1060,11 @@@
  		s.next_in = (void *)dat->buf;
  		s.avail_in = dat->len;
  	}
 -	s.avail_out = deflateBound(&s, s.avail_in);
 +	s.avail_out = git_deflate_bound(&s, s.avail_in);
  	s.next_out = out = xmalloc(s.avail_out);
 -	while (deflate(&s, Z_FINISH) == Z_OK)
 -		/* nothing */;
 -	deflateEnd(&s);
 +	while (git_deflate(&s, Z_FINISH) == Z_OK)
 +		; /* nothing */
 +	git_deflate_end(&s);
  
  	/* Determine if we should auto-checkpoint. */
  	if ((max_packsize && (pack_size + 60 + s.total_out) > max_packsize)
@@@ -1079,14 -1079,14 +1080,14 @@@
  			delta = NULL;
  
  			memset(&s, 0, sizeof(s));
 -			deflateInit(&s, pack_compression_level);
 +			git_deflate_init(&s, pack_compression_level);
  			s.next_in = (void *)dat->buf;
  			s.avail_in = dat->len;
 -			s.avail_out = deflateBound(&s, s.avail_in);
 +			s.avail_out = git_deflate_bound(&s, s.avail_in);
  			s.next_out = out = xrealloc(out, s.avail_out);
 -			while (deflate(&s, Z_FINISH) == Z_OK)
 -				/* nothing */;
 -			deflateEnd(&s);
 +			while (git_deflate(&s, Z_FINISH) == Z_OK)
 +				; /* nothing */
 +			git_deflate_end(&s);
  		}
  	}
  
@@@ -1164,7 -1164,7 +1165,7 @@@ static void stream_blob(uintmax_t len, 
  	off_t offset;
  	git_SHA_CTX c;
  	git_SHA_CTX pack_file_ctx;
 -	z_stream s;
 +	git_zstream s;
  	int status = Z_OK;
  
  	/* Determine if we should auto-checkpoint. */
@@@ -1188,7 -1188,7 +1189,7 @@@
  	crc32_begin(pack_file);
  
  	memset(&s, 0, sizeof(s));
 -	deflateInit(&s, pack_compression_level);
 +	git_deflate_init(&s, pack_compression_level);
  
  	hdrlen = encode_in_pack_object_header(OBJ_BLOB, len, out_buf);
  	if (out_sz <= hdrlen)
@@@ -1210,7 -1210,7 +1211,7 @@@
  			len -= n;
  		}
  
 -		status = deflate(&s, len ? 0 : Z_FINISH);
 +		status = git_deflate(&s, len ? 0 : Z_FINISH);
  
  		if (!s.avail_out || status == Z_STREAM_END) {
  			size_t n = s.next_out - out_buf;
@@@ -1229,7 -1229,7 +1230,7 @@@
  			die("unexpected deflate failure: %d", status);
  		}
  	}
 -	deflateEnd(&s);
 +	git_deflate_end(&s);
  	git_SHA1_Final(sha1, &c);
  
  	if (sha1out)
@@@ -3140,6 -3140,8 +3141,8 @@@ static int parse_one_feature(const cha
  		relative_marks_paths = 1;
  	} else if (!strcmp(feature, "no-relative-marks")) {
  		relative_marks_paths = 0;
+ 	} else if (!strcmp(feature, "done")) {
+ 		require_explicit_termination = 1;
  	} else if (!strcmp(feature, "force")) {
  		force_update = 1;
  	} else if (!strcmp(feature, "notes") || !strcmp(feature, "ls")) {
@@@ -3196,10 -3198,10 +3199,10 @@@ static int git_pack_config(const char *
  		return 0;
  	}
  	if (!strcmp(k, "pack.indexversion")) {
 -		pack_idx_default_version = git_config_int(k, v);
 -		if (pack_idx_default_version > 2)
 +		pack_idx_opts.version = git_config_int(k, v);
 +		if (pack_idx_opts.version > 2)
  			die("bad pack.indexversion=%"PRIu32,
 -			    pack_idx_default_version);
 +			    pack_idx_opts.version);
  		return 0;
  	}
  	if (!strcmp(k, "pack.packsizelimit")) {
@@@ -3253,7 -3255,6 +3256,7 @@@ int main(int argc, const char **argv
  		usage(fast_import_usage);
  
  	setup_git_directory();
 +	reset_pack_idx_option(&pack_idx_opts);
  	git_config(git_pack_config, NULL);
  	if (!pack_compression_seen && core_compression_seen)
  		pack_compression_level = core_compression_level;
@@@ -3290,6 -3291,8 +3293,8 @@@
  			parse_reset_branch();
  		else if (!strcmp("checkpoint", command_buf.buf))
  			parse_checkpoint();
+ 		else if (!strcmp("done", command_buf.buf))
+ 			break;
  		else if (!prefixcmp(command_buf.buf, "progress "))
  			parse_progress();
  		else if (!prefixcmp(command_buf.buf, "feature "))
@@@ -3309,6 -3312,9 +3314,9 @@@
  	if (!seen_data_command)
  		parse_argv();
  
+ 	if (require_explicit_termination && feof(stdin))
+ 		die("stream ends early");
+ 
  	end_packfile();
  
  	dump_branches();
diff --combined remote-curl.c
index 69831e931a,7b3c113b09..b8cf45a7dd
--- a/remote-curl.c
+++ b/remote-curl.c
@@@ -227,8 -227,6 +227,8 @@@ static struct ref *parse_info_refs(stru
  		if (data[i] == '\t')
  			mid = &data[i];
  		if (data[i] == '\n') {
 +			if (mid - start != 40)
 +				die("%sinfo/refs not valid: is this a git repository?", url);
  			data[i] = 0;
  			ref_name = mid + 1;
  			ref = xmalloc(sizeof(struct ref) +
@@@ -473,12 -471,16 +473,12 @@@ static int post_rpc(struct rpc_state *r
  		 * the transfer time.
  		 */
  		size_t size;
 -		z_stream stream;
 +		git_zstream stream;
  		int ret;
  
  		memset(&stream, 0, sizeof(stream));
 -		ret = deflateInit2(&stream, Z_BEST_COMPRESSION,
 -				Z_DEFLATED, (15 + 16),
 -				8, Z_DEFAULT_STRATEGY);
 -		if (ret != Z_OK)
 -			die("cannot deflate request; zlib init error %d", ret);
 -		size = deflateBound(&stream, rpc->len);
 +		git_deflate_init_gzip(&stream, Z_BEST_COMPRESSION);
 +		size = git_deflate_bound(&stream, rpc->len);
  		gzip_body = xmalloc(size);
  
  		stream.next_in = (unsigned char *)rpc->buf;
@@@ -486,11 -488,11 +486,11 @@@
  		stream.next_out = (unsigned char *)gzip_body;
  		stream.avail_out = size;
  
 -		ret = deflate(&stream, Z_FINISH);
 +		ret = git_deflate(&stream, Z_FINISH);
  		if (ret != Z_STREAM_END)
  			die("cannot deflate request; zlib deflate error %d", ret);
  
 -		ret = deflateEnd(&stream);
 +		ret = git_deflate_end_gently(&stream);
  		if (ret != Z_OK)
  			die("cannot deflate request; zlib end error %d", ret);
  
@@@ -809,21 -811,19 +809,21 @@@ static void parse_push(struct strbuf *b
  
  		strbuf_reset(buf);
  		if (strbuf_getline(buf, stdin, '\n') == EOF)
 -			return;
 +			goto free_specs;
  		if (!*buf->buf)
  			break;
  	} while (1);
  
  	if (push(nr_spec, specs))
  		exit(128); /* error already reported */
 -	for (i = 0; i < nr_spec; i++)
 -		free(specs[i]);
 -	free(specs);
  
  	printf("\n");
  	fflush(stdout);
 +
 + free_specs:
 +	for (i = 0; i < nr_spec; i++)
 +		free(specs[i]);
 +	free(specs);
  }
  
  int main(int argc, const char **argv)
@@@ -855,7 -855,14 +855,14 @@@
  	http_init(remote);
  
  	do {
- 		if (strbuf_getline(&buf, stdin, '\n') == EOF)
+ 		if (strbuf_getline(&buf, stdin, '\n') == EOF) {
+ 			if (ferror(stdin))
+ 				fprintf(stderr, "Error reading command stream\n");
+ 			else
+ 				fprintf(stderr, "Unexpected end of command stream\n");
+ 			return 1;
+ 		}
+ 		if (buf.len == 0)
  			break;
  		if (!prefixcmp(buf.buf, "fetch ")) {
  			if (nongit)
@@@ -895,6 -902,7 +902,7 @@@
  			printf("\n");
  			fflush(stdout);
  		} else {
+ 			fprintf(stderr, "Unknown command '%s'\n", buf.buf);
  			return 1;
  		}
  		strbuf_reset(&buf);
diff --combined t/t9300-fast-import.sh
index 2a53640c5b,526231bbeb..f256475020
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@@ -1893,7 -1893,7 +1893,7 @@@ test_expect_success 
      test_cmp marks.out marks.new'
  
  cat >input <<EOF
 -feature import-marks=nonexistant.marks
 +feature import-marks=nonexistent.marks
  feature export-marks=marks.new
  EOF
  
@@@ -1904,7 -1904,7 +1904,7 @@@ test_expect_success 
  
  
  cat >input <<EOF
 -feature import-marks=nonexistant.marks
 +feature import-marks=nonexistent.marks
  feature export-marks=combined.marks
  EOF
  
@@@ -2197,6 -2197,48 +2197,48 @@@ test_expect_success 'R: quiet option re
      test_cmp empty output
  '
  
+ test_expect_success 'R: feature done means terminating "done" is mandatory' '
+ 	echo feature done | test_must_fail git fast-import &&
+ 	test_must_fail git fast-import --done </dev/null
+ '
+ 
+ test_expect_success 'R: terminating "done" with trailing gibberish is ok' '
+ 	git fast-import <<-\EOF &&
+ 	feature done
+ 	done
+ 	trailing gibberish
+ 	EOF
+ 	git fast-import <<-\EOF
+ 	done
+ 	more trailing gibberish
+ 	EOF
+ '
+ 
+ test_expect_success 'R: terminating "done" within commit' '
+ 	cat >expect <<-\EOF &&
+ 	OBJID
+ 	:000000 100644 OBJID OBJID A	hello.c
+ 	:000000 100644 OBJID OBJID A	hello2.c
+ 	EOF
+ 	git fast-import <<-EOF &&
+ 	commit refs/heads/done-ends
+ 	committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> $GIT_COMMITTER_DATE
+ 	data <<EOT
+ 	Commit terminated by "done" command
+ 	EOT
+ 	M 100644 inline hello.c
+ 	data <<EOT
+ 	Hello, world.
+ 	EOT
+ 	C hello.c hello2.c
+ 	done
+ 	EOF
+ 	git rev-list done-ends |
+ 	git diff-tree -r --stdin --root --always |
+ 	sed -e "s/$_x40/OBJID/g" >actual &&
+ 	test_cmp expect actual
+ '
+ 
  cat >input <<EOF
  option git non-existing-option
  EOF