From: Junio C Hamano <gitster@pobox.com>
Date: Tue, 8 May 2018 06:59:22 +0000 (+0900)
Subject: Merge branch 'ls/checkout-encoding'
X-Git-Tag: v2.18.0-rc0~95
X-Git-Url: https://git.lorimer.id.au/gitweb.git/diff_plain/1ac0ce4d32ab7a3546e7e84a562625576208c7db?ds=inline;hp=-c

Merge branch 'ls/checkout-encoding'

The new "checkout-encoding" attribute can ask Git to convert the
contents to the specified encoding when checking out to the working
tree (and the other way around when checking in).

* ls/checkout-encoding:
convert: add round trip check based on 'core.checkRoundtripEncoding'
convert: add tracing for 'working-tree-encoding' attribute
convert: check for detectable errors in UTF encodings
convert: add 'working-tree-encoding' attribute
utf8: add function to detect a missing UTF-16/32 BOM
utf8: add function to detect prohibited UTF-16/32 BOM
utf8: teach same_encoding() alternative UTF encoding names
strbuf: add a case insensitive starts_with()
strbuf: add xstrdup_toupper()
strbuf: remove unnecessary NUL assignment in xstrdup_tolower()
---

1ac0ce4d32ab7a3546e7e84a562625576208c7db
diff --combined Documentation/config.txt
index cb4c93f74a,4859a7dedd..8213c1236b
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@@ -41,13 -41,11 +41,13 @@@ in the section header, like in the exam
  --------
  
  Subsection names are case sensitive and can contain any characters except
 -newline (doublequote `"` and backslash can be included by escaping them
 -as `\"` and `\\`, respectively).  Section headers cannot span multiple
 -lines.  Variables may belong directly to a section or to a given subsection.
 -You can have `[section]` if you have `[section "subsection"]`, but you
 -don't need to.
 +newline and the null byte. Doublequote `"` and backslash can be included
 +by escaping them as `\"` and `\\`, respectively. Backslashes preceding
 +other characters are dropped when reading; for example, `\t` is read as
 +`t` and `\0` is read as `0` Section headers cannot span multiple lines.
 +Variables may belong directly to a section or to a given subsection. You
 +can have `[section]` if you have `[section "subsection"]`, but you don't
 +need to.
  
  There is also a deprecated `[section.subsection]` syntax. With this
  syntax, the subsection name is converted to lower-case and is also
@@@ -530,6 -528,12 +530,12 @@@ core.autocrlf:
  	This variable can be set to 'input',
  	in which case no output conversion is performed.
  
+ core.checkRoundtripEncoding::
+ 	A comma and/or whitespace separated list of encodings that Git
+ 	performs UTF-8 round trip checks on if they are used in an
+ 	`working-tree-encoding` attribute (see linkgit:gitattributes[5]).
+ 	The default value is `SHIFT-JIS`.
+ 
  core.symlinks::
  	If false, symbolic links are checked out as small plain files that
  	contain the link text. linkgit:git-update-index[1] and
@@@ -898,10 -902,6 +904,10 @@@ core.notesRef:
  This setting defaults to "refs/notes/commits", and it can be overridden by
  the `GIT_NOTES_REF` environment variable.  See linkgit:git-notes[1].
  
 +core.commitGraph::
 +	Enable git commit graph feature. Allows reading from the
 +	commit-graph file.
 +
  core.sparseCheckout::
  	Enable "sparse checkout" feature. See section "Sparse checkout" in
  	linkgit:git-read-tree[1] for more information.
@@@ -1402,16 -1402,7 +1408,16 @@@ fetch.unpackLimit:
  
  fetch.prune::
  	If true, fetch will automatically behave as if the `--prune`
 -	option was given on the command line.  See also `remote.<name>.prune`.
 +	option was given on the command line.  See also `remote.<name>.prune`
 +	and the PRUNING section of linkgit:git-fetch[1].
 +
 +fetch.pruneTags::
 +	If true, fetch will automatically behave as if the
 +	`refs/tags/*:refs/tags/*` refspec was provided when pruning,
 +	if not set already. This allows for setting both this option
 +	and `fetch.prune` to maintain a 1=1 mapping to upstream
 +	refs. See also `remote.<name>.pruneTags` and the PRUNING
 +	section of linkgit:git-fetch[1].
  
  fetch.output::
  	Control how ref update status is printed. Valid values are
@@@ -1961,7 -1952,6 +1967,7 @@@ http.sslVersion:
  	- tlsv1.0
  	- tlsv1.1
  	- tlsv1.2
 +	- tlsv1.3
  
  +
  Can be overridden by the `GIT_SSL_VERSION` environment variable.
@@@ -2959,15 -2949,6 +2965,15 @@@ remote.<name>.prune:
  	remote (as if the `--prune` option was given on the command line).
  	Overrides `fetch.prune` settings, if any.
  
 +remote.<name>.pruneTags::
 +	When set to true, fetching from this remote by default will also
 +	remove any local tags that no longer exist on the remote if pruning
 +	is activated in general via `remote.<name>.prune`, `fetch.prune` or
 +	`--prune`. Overrides `fetch.pruneTags` settings, if any.
 ++
 +See also `remote.<name>.prune` and the PRUNING section of
 +linkgit:git-fetch[1].
 +
  remotes.<group>::
  	The list of remotes which are fetched by "git remote update
  	<group>".  See linkgit:git-remote[1].
@@@ -3233,8 -3214,7 +3239,8 @@@ submodule.active:
  
  submodule.recurse::
  	Specifies if commands recurse into submodules by default. This
 -	applies to all commands that have a `--recurse-submodules` option.
 +	applies to all commands that have a `--recurse-submodules` option,
 +	except `clone`.
  	Defaults to false.
  
  submodule.fetchJobs::
@@@ -3367,10 -3347,6 +3373,10 @@@ uploadpack.packObjectsHook:
  	was run. I.e., `upload-pack` will feed input intended for
  	`pack-objects` to the hook, and expects a completed packfile on
  	stdout.
 +
 +uploadpack.allowFilter::
 +	If this option is set, `upload-pack` will support partial
 +	clone and partial fetch object filtering.
  +
  Note that this configuration variable is ignored if it is seen in the
  repository-level config (this is a safety measure against fetching from
diff --combined Documentation/gitattributes.txt
index 1094fe2b5b,aa3deae392..ee210be3ec
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@@ -56,16 -56,9 +56,16 @@@ Unspecified:
  
  When more than one pattern matches the path, a later line
  overrides an earlier line.  This overriding is done per
 -attribute.  The rules how the pattern matches paths are the
 -same as in `.gitignore` files; see linkgit:gitignore[5].
 -Unlike `.gitignore`, negative patterns are forbidden.
 +attribute.
 +
 +The rules by which the pattern matches paths are the same as in
 +`.gitignore` files (see linkgit:gitignore[5]), with a few exceptions:
 +
 +  - negative patterns are forbidden
 +
 +  - patterns that match a directory do not recursively match paths
 +    inside that directory (so using the trailing-slash `path/` syntax is
 +    pointless in an attributes file; use `path/**` instead)
  
  When deciding what attributes are assigned to a path, Git
  consults `$GIT_DIR/info/attributes` file (which has the highest
@@@ -279,6 -272,94 +279,94 @@@ few exceptions.  Even though..
    catch potential problems early, safety triggers.
  
  
+ `working-tree-encoding`
+ ^^^^^^^^^^^^^^^^^^^^^^^
+ 
+ Git recognizes files encoded in ASCII or one of its supersets (e.g.
+ UTF-8, ISO-8859-1, ...) as text files. Files encoded in certain other
+ encodings (e.g. UTF-16) are interpreted as binary and consequently
+ built-in Git text processing tools (e.g. 'git diff') as well as most Git
+ web front ends do not visualize the contents of these files by default.
+ 
+ In these cases you can tell Git the encoding of a file in the working
+ directory with the `working-tree-encoding` attribute. If a file with this
+ attribute is added to Git, then Git reencodes the content from the
+ specified encoding to UTF-8. Finally, Git stores the UTF-8 encoded
+ content in its internal data structure (called "the index"). On checkout
+ the content is reencoded back to the specified encoding.
+ 
+ Please note that using the `working-tree-encoding` attribute may have a
+ number of pitfalls:
+ 
+ - Alternative Git implementations (e.g. JGit or libgit2) and older Git
+   versions (as of March 2018) do not support the `working-tree-encoding`
+   attribute. If you decide to use the `working-tree-encoding` attribute
+   in your repository, then it is strongly recommended to ensure that all
+   clients working with the repository support it.
+ 
+   For example, Microsoft Visual Studio resources files (`*.rc`) or
+   PowerShell script files (`*.ps1`) are sometimes encoded in UTF-16.
+   If you declare `*.ps1` as files as UTF-16 and you add `foo.ps1` with
+   a `working-tree-encoding` enabled Git client, then `foo.ps1` will be
+   stored as UTF-8 internally. A client without `working-tree-encoding`
+   support will checkout `foo.ps1` as UTF-8 encoded file. This will
+   typically cause trouble for the users of this file.
+ 
+   If a Git client, that does not support the `working-tree-encoding`
+   attribute, adds a new file `bar.ps1`, then `bar.ps1` will be
+   stored "as-is" internally (in this example probably as UTF-16).
+   A client with `working-tree-encoding` support will interpret the
+   internal contents as UTF-8 and try to convert it to UTF-16 on checkout.
+   That operation will fail and cause an error.
+ 
+ - Reencoding content to non-UTF encodings can cause errors as the
+   conversion might not be UTF-8 round trip safe. If you suspect your
+   encoding to not be round trip safe, then add it to
+   `core.checkRoundtripEncoding` to make Git check the round trip
+   encoding (see linkgit:git-config[1]). SHIFT-JIS (Japanese character
+   set) is known to have round trip issues with UTF-8 and is checked by
+   default.
+ 
+ - Reencoding content requires resources that might slow down certain
+   Git operations (e.g 'git checkout' or 'git add').
+ 
+ Use the `working-tree-encoding` attribute only if you cannot store a file
+ in UTF-8 encoding and if you want Git to be able to process the content
+ as text.
+ 
+ As an example, use the following attributes if your '*.ps1' files are
+ UTF-16 encoded with byte order mark (BOM) and you want Git to perform
+ automatic line ending conversion based on your platform.
+ 
+ ------------------------
+ *.ps1		text working-tree-encoding=UTF-16
+ ------------------------
+ 
+ Use the following attributes if your '*.ps1' files are UTF-16 little
+ endian encoded without BOM and you want Git to use Windows line endings
+ in the working directory. Please note, it is highly recommended to
+ explicitly define the line endings with `eol` if the `working-tree-encoding`
+ attribute is used to avoid ambiguity.
+ 
+ ------------------------
+ *.ps1		text working-tree-encoding=UTF-16LE eol=CRLF
+ ------------------------
+ 
+ You can get a list of all available encodings on your platform with the
+ following command:
+ 
+ ------------------------
+ iconv --list
+ ------------------------
+ 
+ If you do not know the encoding of a file, then you can use the `file`
+ command to guess the encoding:
+ 
+ ------------------------
+ file foo.ps1
+ ------------------------
+ 
+ 
  `ident`
  ^^^^^^^
  
@@@ -399,14 -480,46 +487,14 @@@ Long Running Filter Proces
  If the filter command (a string value) is defined via
  `filter.<driver>.process` then Git can process all blobs with a
  single filter invocation for the entire life of a single Git
 -command. This is achieved by using a packet format (pkt-line,
 -see technical/protocol-common.txt) based protocol over standard
 -input and standard output as follows. All packets, except for the
 -"*CONTENT" packets and the "0000" flush packet, are considered
 -text and therefore are terminated by a LF.
 -
 -Git starts the filter when it encounters the first file
 -that needs to be cleaned or smudged. After the filter started
 -Git sends a welcome message ("git-filter-client"), a list of supported
 -protocol version numbers, and a flush packet. Git expects to read a welcome
 -response message ("git-filter-server"), exactly one protocol version number
 -from the previously sent list, and a flush packet. All further
 -communication will be based on the selected version. The remaining
 -protocol description below documents "version=2". Please note that
 -"version=42" in the example below does not exist and is only there
 -to illustrate how the protocol would look like with more than one
 -version.
 -
 -After the version negotiation Git sends a list of all capabilities that
 -it supports and a flush packet. Git expects to read a list of desired
 -capabilities, which must be a subset of the supported capabilities list,
 -and a flush packet as response:
 -------------------------
 -packet:          git> git-filter-client
 -packet:          git> version=2
 -packet:          git> version=42
 -packet:          git> 0000
 -packet:          git< git-filter-server
 -packet:          git< version=2
 -packet:          git< 0000
 -packet:          git> capability=clean
 -packet:          git> capability=smudge
 -packet:          git> capability=not-yet-invented
 -packet:          git> 0000
 -packet:          git< capability=clean
 -packet:          git< capability=smudge
 -packet:          git< 0000
 -------------------------
 -Supported filter capabilities in version 2 are "clean", "smudge",
 -and "delay".
 +command. This is achieved by using the long-running process protocol
 +(described in technical/long-running-process-protocol.txt).
 +
 +When Git encounters the first file that needs to be cleaned or smudged,
 +it starts the filter and performs the handshake. In the handshake, the
 +welcome message sent by Git is "git-filter-client", only version 2 is
 +suppported, and the supported capabilities are "clean", "smudge", and
 +"delay".
  
  Afterwards Git sends a list of "key=value" pairs terminated with
  a flush packet. The list will contain at least the filter command
@@@ -492,6 -605,12 +580,6 @@@ the protocol then Git will stop the fil
  with the next file that needs to be processed. Depending on the
  `filter.<driver>.required` flag Git will interpret that as error.
  
 -After the filter has processed a command it is expected to wait for
 -a "key=value" list containing the next command. Git will close
 -the command pipe on exit. The filter is expected to detect EOF
 -and exit gracefully on its own. Git will wait until the filter
 -process has stopped.
 -
  Delay
  ^^^^^
  
@@@ -721,8 -840,6 +809,8 @@@ patterns are available
  
  - `fountain` suitable for Fountain documents.
  
 +- `golang` suitable for source code in the Go language.
 +
  - `html` suitable for HTML/XHTML documents.
  
  - `java` suitable for source code in the Java language.
diff --combined config.c
index 40d46b3325,d0ada9fcd4..9f8ca18f7b
--- a/config.c
+++ b/config.c
@@@ -9,7 -9,7 +9,7 @@@
  #include "config.h"
  #include "repository.h"
  #include "lockfile.h"
 -#include "exec_cmd.h"
 +#include "exec-cmd.h"
  #include "strbuf.h"
  #include "quote.h"
  #include "hashmap.h"
@@@ -653,45 -653,7 +653,45 @@@ static int get_base_var(struct strbuf *
  	}
  }
  
 -static int git_parse_source(config_fn_t fn, void *data)
 +struct parse_event_data {
 +	enum config_event_t previous_type;
 +	size_t previous_offset;
 +	const struct config_options *opts;
 +};
 +
 +static int do_event(enum config_event_t type, struct parse_event_data *data)
 +{
 +	size_t offset;
 +
 +	if (!data->opts || !data->opts->event_fn)
 +		return 0;
 +
 +	if (type == CONFIG_EVENT_WHITESPACE &&
 +	    data->previous_type == type)
 +		return 0;
 +
 +	offset = cf->do_ftell(cf);
 +	/*
 +	 * At EOF, the parser always "inserts" an extra '\n', therefore
 +	 * the end offset of the event is the current file position, otherwise
 +	 * we will already have advanced to the next event.
 +	 */
 +	if (type != CONFIG_EVENT_EOF)
 +		offset--;
 +
 +	if (data->previous_type != CONFIG_EVENT_EOF &&
 +	    data->opts->event_fn(data->previous_type, data->previous_offset,
 +				 offset, data->opts->event_fn_data) < 0)
 +		return -1;
 +
 +	data->previous_type = type;
 +	data->previous_offset = offset;
 +
 +	return 0;
 +}
 +
 +static int git_parse_source(config_fn_t fn, void *data,
 +			    const struct config_options *opts)
  {
  	int comment = 0;
  	int baselen = 0;
@@@ -702,15 -664,8 +702,15 @@@
  	/* U+FEFF Byte Order Mark in UTF8 */
  	const char *bomptr = utf8_bom;
  
 +	/* For the parser event callback */
 +	struct parse_event_data event_data = {
 +		CONFIG_EVENT_EOF, 0, opts
 +	};
 +
  	for (;;) {
 -		int c = get_next_char();
 +		int c;
 +
 +		c = get_next_char();
  		if (bomptr && *bomptr) {
  			/* We are at the file beginning; skip UTF8-encoded BOM
  			 * if present. Sane editors won't put this in on their
@@@ -727,33 -682,18 +727,33 @@@
  			}
  		}
  		if (c == '\n') {
 -			if (cf->eof)
 +			if (cf->eof) {
 +				if (do_event(CONFIG_EVENT_EOF, &event_data) < 0)
 +					return -1;
  				return 0;
 +			}
 +			if (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)
 +				return -1;
  			comment = 0;
  			continue;
  		}
 -		if (comment || isspace(c))
 +		if (comment)
 +			continue;
 +		if (isspace(c)) {
 +			if (do_event(CONFIG_EVENT_WHITESPACE, &event_data) < 0)
 +					return -1;
  			continue;
 +		}
  		if (c == '#' || c == ';') {
 +			if (do_event(CONFIG_EVENT_COMMENT, &event_data) < 0)
 +					return -1;
  			comment = 1;
  			continue;
  		}
  		if (c == '[') {
 +			if (do_event(CONFIG_EVENT_SECTION, &event_data) < 0)
 +					return -1;
 +
  			/* Reset prior to determining a new stem */
  			strbuf_reset(var);
  			if (get_base_var(var) < 0 || var->len < 1)
@@@ -764,10 -704,6 +764,10 @@@
  		}
  		if (!isalpha(c))
  			break;
 +
 +		if (do_event(CONFIG_EVENT_ENTRY, &event_data) < 0)
 +			return -1;
 +
  		/*
  		 * Truncate the var name back to the section header
  		 * stem prior to grabbing the suffix part of the name
@@@ -779,9 -715,6 +779,9 @@@
  			break;
  	}
  
 +	if (do_event(CONFIG_EVENT_ERROR, &event_data) < 0)
 +		return -1;
 +
  	switch (cf->origin_type) {
  	case CONFIG_ORIGIN_BLOB:
  		error_msg = xstrfmt(_("bad config line %d in blob %s"),
@@@ -1239,6 -1172,11 +1239,11 @@@ static int git_default_core_config(cons
  		return 0;
  	}
  
+ 	if (!strcmp(var, "core.checkroundtripencoding")) {
+ 		check_roundtrip_encoding = xstrdup(value);
+ 		return 0;
+ 	}
+ 
  	if (!strcmp(var, "core.notesref")) {
  		notes_ref_name = xstrdup(value);
  		return 0;
@@@ -1293,11 -1231,6 +1298,11 @@@
  		return 0;
  	}
  
 +	if (!strcmp(var, "core.commitgraph")) {
 +		core_commit_graph = git_config_bool(var, value);
 +		return 0;
 +	}
 +
  	if (!strcmp(var, "core.sparsecheckout")) {
  		core_apply_sparse_checkout = git_config_bool(var, value);
  		return 0;
@@@ -1326,11 -1259,6 +1331,11 @@@
  		return 0;
  	}
  
 +	if (!strcmp(var, "core.partialclonefilter")) {
 +		return git_config_string(&core_partial_clone_filter_default,
 +					 var, value);
 +	}
 +
  	/* Add other config variables here and to Documentation/config.txt. */
  	return 0;
  }
@@@ -1470,8 -1398,7 +1475,8 @@@ int git_default_config(const char *var
   * fgetc, ungetc, ftell of top need to be initialized before calling
   * this function.
   */
 -static int do_config_from(struct config_source *top, config_fn_t fn, void *data)
 +static int do_config_from(struct config_source *top, config_fn_t fn, void *data,
 +			  const struct config_options *opts)
  {
  	int ret;
  
@@@ -1483,7 -1410,7 +1488,7 @@@
  	strbuf_init(&top->var, 1024);
  	cf = top;
  
 -	ret = git_parse_source(fn, data);
 +	ret = git_parse_source(fn, data, opts);
  
  	/* pop config-file parsing state stack */
  	strbuf_release(&top->value);
@@@ -1496,10 -1423,9 +1501,10 @@@
  static int do_config_from_file(config_fn_t fn,
  		const enum config_origin_type origin_type,
  		const char *name, const char *path, FILE *f,
 -		void *data)
 +		void *data, const struct config_options *opts)
  {
  	struct config_source top;
 +	int ret;
  
  	top.u.file = f;
  	top.origin_type = origin_type;
@@@ -1510,39 -1436,29 +1515,39 @@@
  	top.do_ungetc = config_file_ungetc;
  	top.do_ftell = config_file_ftell;
  
 -	return do_config_from(&top, fn, data);
 +	flockfile(f);
 +	ret = do_config_from(&top, fn, data, opts);
 +	funlockfile(f);
 +	return ret;
  }
  
  static int git_config_from_stdin(config_fn_t fn, void *data)
  {
 -	return do_config_from_file(fn, CONFIG_ORIGIN_STDIN, "", NULL, stdin, data);
 +	return do_config_from_file(fn, CONFIG_ORIGIN_STDIN, "", NULL, stdin,
 +				   data, NULL);
  }
  
 -int git_config_from_file(config_fn_t fn, const char *filename, void *data)
 +int git_config_from_file_with_options(config_fn_t fn, const char *filename,
 +				      void *data,
 +				      const struct config_options *opts)
  {
  	int ret = -1;
  	FILE *f;
  
  	f = fopen_or_warn(filename, "r");
  	if (f) {
 -		flockfile(f);
 -		ret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename, filename, f, data);
 -		funlockfile(f);
 +		ret = do_config_from_file(fn, CONFIG_ORIGIN_FILE, filename,
 +					  filename, f, data, opts);
  		fclose(f);
  	}
  	return ret;
  }
  
 +int git_config_from_file(config_fn_t fn, const char *filename, void *data)
 +{
 +	return git_config_from_file_with_options(fn, filename, data, NULL);
 +}
 +
  int git_config_from_mem(config_fn_t fn, const enum config_origin_type origin_type,
  			const char *name, const char *buf, size_t len, void *data)
  {
@@@ -1559,7 -1475,7 +1564,7 @@@
  	top.do_ungetc = config_buf_ungetc;
  	top.do_ftell = config_buf_ftell;
  
 -	return do_config_from(&top, fn, data);
 +	return do_config_from(&top, fn, data, NULL);
  }
  
  int git_config_from_blob_oid(config_fn_t fn,
@@@ -1572,7 -1488,7 +1577,7 @@@
  	unsigned long size;
  	int ret;
  
 -	buf = read_sha1_file(oid->hash, &type, &size);
 +	buf = read_object_file(oid, &type, &size);
  	if (!buf)
  		return error("unable to load config blob object '%s'", name);
  	if (type != OBJ_BLOB) {
@@@ -2303,98 -2219,96 +2308,98 @@@ void git_die_config(const char *key, co
   * Find all the stuff for git_config_set() below.
   */
  
 -static struct {
 +struct config_store_data {
  	int baselen;
  	char *key;
  	int do_not_match;
  	regex_t *value_regex;
  	int multi_replace;
 -	size_t *offset;
 -	unsigned int offset_alloc;
 -	enum { START, SECTION_SEEN, SECTION_END_SEEN, KEY_SEEN } state;
 -	unsigned int seen;
 -} store;
 +	struct {
 +		size_t begin, end;
 +		enum config_event_t type;
 +		int is_keys_section;
 +	} *parsed;
 +	unsigned int parsed_nr, parsed_alloc, *seen, seen_nr, seen_alloc;
 +	unsigned int key_seen:1, section_seen:1, is_keys_section:1;
 +};
  
 -static int matches(const char *key, const char *value)
 +static int matches(const char *key, const char *value,
 +		   const struct config_store_data *store)
  {
 -	if (strcmp(key, store.key))
 +	if (strcmp(key, store->key))
  		return 0; /* not ours */
 -	if (!store.value_regex)
 +	if (!store->value_regex)
  		return 1; /* always matches */
 -	if (store.value_regex == CONFIG_REGEX_NONE)
 +	if (store->value_regex == CONFIG_REGEX_NONE)
  		return 0; /* never matches */
  
 -	return store.do_not_match ^
 -		(value && !regexec(store.value_regex, value, 0, NULL, 0));
 +	return store->do_not_match ^
 +		(value && !regexec(store->value_regex, value, 0, NULL, 0));
 +}
 +
 +static int store_aux_event(enum config_event_t type,
 +			   size_t begin, size_t end, void *data)
 +{
 +	struct config_store_data *store = data;
 +
 +	ALLOC_GROW(store->parsed, store->parsed_nr + 1, store->parsed_alloc);
 +	store->parsed[store->parsed_nr].begin = begin;
 +	store->parsed[store->parsed_nr].end = end;
 +	store->parsed[store->parsed_nr].type = type;
 +
 +	if (type == CONFIG_EVENT_SECTION) {
 +		if (cf->var.len < 2 || cf->var.buf[cf->var.len - 1] != '.')
 +			BUG("Invalid section name '%s'", cf->var.buf);
 +
 +		/* Is this the section we were looking for? */
 +		store->is_keys_section =
 +			store->parsed[store->parsed_nr].is_keys_section =
 +			cf->var.len - 1 == store->baselen &&
 +			!strncasecmp(cf->var.buf, store->key, store->baselen);
 +		if (store->is_keys_section) {
 +			store->section_seen = 1;
 +			ALLOC_GROW(store->seen, store->seen_nr + 1,
 +				   store->seen_alloc);
 +			store->seen[store->seen_nr] = store->parsed_nr;
 +		}
 +	}
 +
 +	store->parsed_nr++;
 +
 +	return 0;
  }
  
  static int store_aux(const char *key, const char *value, void *cb)
  {
 -	const char *ep;
 -	size_t section_len;
 +	struct config_store_data *store = cb;
  
 -	switch (store.state) {
 -	case KEY_SEEN:
 -		if (matches(key, value)) {
 -			if (store.seen == 1 && store.multi_replace == 0) {
 +	if (store->key_seen) {
 +		if (matches(key, value, store)) {
 +			if (store->seen_nr == 1 && store->multi_replace == 0) {
  				warning(_("%s has multiple values"), key);
  			}
  
 -			ALLOC_GROW(store.offset, store.seen + 1,
 -				   store.offset_alloc);
 +			ALLOC_GROW(store->seen, store->seen_nr + 1,
 +				   store->seen_alloc);
  
 -			store.offset[store.seen] = cf->do_ftell(cf);
 -			store.seen++;
 +			store->seen[store->seen_nr] = store->parsed_nr;
 +			store->seen_nr++;
  		}
 -		break;
 -	case SECTION_SEEN:
 +	} else if (store->is_keys_section) {
  		/*
 -		 * What we are looking for is in store.key (both
 -		 * section and var), and its section part is baselen
 -		 * long.  We found key (again, both section and var).
 -		 * We would want to know if this key is in the same
 -		 * section as what we are looking for.  We already
 -		 * know we are in the same section as what should
 -		 * hold store.key.
 +		 * Do not increment matches yet: this may not be a match, but we
 +		 * are in the desired section.
  		 */
 -		ep = strrchr(key, '.');
 -		section_len = ep - key;
 -
 -		if ((section_len != store.baselen) ||
 -		    memcmp(key, store.key, section_len+1)) {
 -			store.state = SECTION_END_SEEN;
 -			break;
 -		}
 +		ALLOC_GROW(store->seen, store->seen_nr + 1, store->seen_alloc);
 +		store->seen[store->seen_nr] = store->parsed_nr;
 +		store->section_seen = 1;
  
 -		/*
 -		 * Do not increment matches: this is no match, but we
 -		 * just made sure we are in the desired section.
 -		 */
 -		ALLOC_GROW(store.offset, store.seen + 1,
 -			   store.offset_alloc);
 -		store.offset[store.seen] = cf->do_ftell(cf);
 -		/* fallthru */
 -	case SECTION_END_SEEN:
 -	case START:
 -		if (matches(key, value)) {
 -			ALLOC_GROW(store.offset, store.seen + 1,
 -				   store.offset_alloc);
 -			store.offset[store.seen] = cf->do_ftell(cf);
 -			store.state = KEY_SEEN;
 -			store.seen++;
 -		} else {
 -			if (strrchr(key, '.') - key == store.baselen &&
 -			      !strncmp(key, store.key, store.baselen)) {
 -					store.state = SECTION_SEEN;
 -					ALLOC_GROW(store.offset,
 -						   store.seen + 1,
 -						   store.offset_alloc);
 -					store.offset[store.seen] = cf->do_ftell(cf);
 -			}
 +		if (matches(key, value, store)) {
 +			store->seen_nr++;
 +			store->key_seen = 1;
  		}
  	}
 +
  	return 0;
  }
  
@@@ -2406,33 -2320,31 +2411,33 @@@ static int write_error(const char *file
  	return 4;
  }
  
 -static struct strbuf store_create_section(const char *key)
 +static struct strbuf store_create_section(const char *key,
 +					  const struct config_store_data *store)
  {
  	const char *dot;
  	int i;
  	struct strbuf sb = STRBUF_INIT;
  
 -	dot = memchr(key, '.', store.baselen);
 +	dot = memchr(key, '.', store->baselen);
  	if (dot) {
  		strbuf_addf(&sb, "[%.*s \"", (int)(dot - key), key);
 -		for (i = dot - key + 1; i < store.baselen; i++) {
 +		for (i = dot - key + 1; i < store->baselen; i++) {
  			if (key[i] == '"' || key[i] == '\\')
  				strbuf_addch(&sb, '\\');
  			strbuf_addch(&sb, key[i]);
  		}
  		strbuf_addstr(&sb, "\"]\n");
  	} else {
 -		strbuf_addf(&sb, "[%.*s]\n", store.baselen, key);
 +		strbuf_addf(&sb, "[%.*s]\n", store->baselen, key);
  	}
  
  	return sb;
  }
  
 -static ssize_t write_section(int fd, const char *key)
 +static ssize_t write_section(int fd, const char *key,
 +			     const struct config_store_data *store)
  {
 -	struct strbuf sb = store_create_section(key);
 +	struct strbuf sb = store_create_section(key, store);
  	ssize_t ret;
  
  	ret = write_in_full(fd, sb.buf, sb.len);
@@@ -2441,12 -2353,11 +2446,12 @@@
  	return ret;
  }
  
 -static ssize_t write_pair(int fd, const char *key, const char *value)
 +static ssize_t write_pair(int fd, const char *key, const char *value,
 +			  const struct config_store_data *store)
  {
  	int i;
  	ssize_t ret;
 -	int length = strlen(key + store.baselen + 1);
 +	int length = strlen(key + store->baselen + 1);
  	const char *quote = "";
  	struct strbuf sb = STRBUF_INIT;
  
@@@ -2466,7 -2377,7 +2471,7 @@@
  		quote = "\"";
  
  	strbuf_addf(&sb, "\t%.*s = %s",
 -		    length, key + store.baselen + 1, quote);
 +		    length, key + store->baselen + 1, quote);
  
  	for (i = 0; value[i]; i++)
  		switch (value[i]) {
@@@ -2492,85 -2403,30 +2497,85 @@@
  	return ret;
  }
  
 -static ssize_t find_beginning_of_line(const char *contents, size_t size,
 -	size_t offset_, int *found_bracket)
 +/*
 + * If we are about to unset the last key(s) in a section, and if there are
 + * no comments surrounding (or included in) the section, we will want to
 + * extend begin/end to remove the entire section.
 + *
 + * Note: the parameter `seen_ptr` points to the index into the store.seen
 + * array.  * This index may be incremented if a section has more than one
 + * entry (which all are to be removed).
 + */
 +static void maybe_remove_section(struct config_store_data *store,
 +				 const char *contents,
 +				 size_t *begin_offset, size_t *end_offset,
 +				 int *seen_ptr)
  {
 -	size_t equal_offset = size, bracket_offset = size;
 -	ssize_t offset;
 +	size_t begin;
 +	int i, seen, section_seen = 0;
  
 -contline:
 -	for (offset = offset_-2; offset > 0
 -			&& contents[offset] != '\n'; offset--)
 -		switch (contents[offset]) {
 -			case '=': equal_offset = offset; break;
 -			case ']': bracket_offset = offset; break;
 +	/*
 +	 * First, ensure that this is the first key, and that there are no
 +	 * comments before the entry nor before the section header.
 +	 */
 +	seen = *seen_ptr;
 +	for (i = store->seen[seen]; i > 0; i--) {
 +		enum config_event_t type = store->parsed[i - 1].type;
 +
 +		if (type == CONFIG_EVENT_COMMENT)
 +			/* There is a comment before this entry or section */
 +			return;
 +		if (type == CONFIG_EVENT_ENTRY) {
 +			if (!section_seen)
 +				/* This is not the section's first entry. */
 +				return;
 +			/* We encountered no comment before the section. */
 +			break;
 +		}
 +		if (type == CONFIG_EVENT_SECTION) {
 +			if (!store->parsed[i - 1].is_keys_section)
 +				break;
 +			section_seen = 1;
  		}
 -	if (offset > 0 && contents[offset-1] == '\\') {
 -		offset_ = offset;
 -		goto contline;
  	}
 -	if (bracket_offset < equal_offset) {
 -		*found_bracket = 1;
 -		offset = bracket_offset+1;
 -	} else
 -		offset++;
 +	begin = store->parsed[i].begin;
  
 -	return offset;
 +	/*
 +	 * Next, make sure that we are removing he last key(s) in the section,
 +	 * and that there are no comments that are possibly about the current
 +	 * section.
 +	 */
 +	for (i = store->seen[seen] + 1; i < store->parsed_nr; i++) {
 +		enum config_event_t type = store->parsed[i].type;
 +
 +		if (type == CONFIG_EVENT_COMMENT)
 +			return;
 +		if (type == CONFIG_EVENT_SECTION) {
 +			if (store->parsed[i].is_keys_section)
 +				continue;
 +			break;
 +		}
 +		if (type == CONFIG_EVENT_ENTRY) {
 +			if (++seen < store->seen_nr &&
 +			    i == store->seen[seen])
 +				/* We want to remove this entry, too */
 +				continue;
 +			/* There is another entry in this section. */
 +			return;
 +		}
 +	}
 +
 +	/*
 +	 * We are really removing the last entry/entries from this section, and
 +	 * there are no enclosed or surrounding comments. Remove the entire,
 +	 * now-empty section.
 +	 */
 +	*seen_ptr = seen;
 +	*begin_offset = begin;
 +	if (i < store->parsed_nr)
 +		*end_offset = store->parsed[i].begin;
 +	else
 +		*end_offset = store->parsed[store->parsed_nr - 1].end;
  }
  
  int git_config_set_in_file_gently(const char *config_filename,
@@@ -2631,9 -2487,6 +2636,9 @@@ int git_config_set_multivar_in_file_gen
  	char *filename_buf = NULL;
  	char *contents = NULL;
  	size_t contents_sz;
 +	struct config_store_data store;
 +
 +	memset(&store, 0, sizeof(store));
  
  	/* parse-key returns negative; flip the sign to feed exit(3) */
  	ret = 0 - git_config_parse_key(key, &store.key, &store.baselen);
@@@ -2676,14 -2529,13 +2681,14 @@@
  		}
  
  		store.key = (char *)key;
 -		if (write_section(fd, key) < 0 ||
 -		    write_pair(fd, key, value) < 0)
 +		if (write_section(fd, key, &store) < 0 ||
 +		    write_pair(fd, key, value, &store) < 0)
  			goto write_err_out;
  	} else {
  		struct stat st;
  		size_t copy_begin, copy_end;
  		int i, new_line = 0;
 +		struct config_options opts;
  
  		if (value_regex == NULL)
  			store.value_regex = NULL;
@@@ -2706,24 -2558,18 +2711,24 @@@
  			}
  		}
  
 -		ALLOC_GROW(store.offset, 1, store.offset_alloc);
 -		store.offset[0] = 0;
 -		store.state = START;
 -		store.seen = 0;
 +		ALLOC_GROW(store.parsed, 1, store.parsed_alloc);
 +		store.parsed[0].end = 0;
 +
 +		memset(&opts, 0, sizeof(opts));
 +		opts.event_fn = store_aux_event;
 +		opts.event_fn_data = &store;
  
  		/*
 -		 * After this, store.offset will contain the *end* offset
 -		 * of the last match, or remain at 0 if no match was found.
 +		 * After this, store.parsed will contain offsets of all the
 +		 * parsed elements, and store.seen will contain a list of
 +		 * matches, as indices into store.parsed.
 +		 *
  		 * As a side effect, we make sure to transform only a valid
  		 * existing config file.
  		 */
 -		if (git_config_from_file(store_aux, config_filename, NULL)) {
 +		if (git_config_from_file_with_options(store_aux,
 +						      config_filename,
 +						      &store, &opts)) {
  			error("invalid config file %s", config_filename);
  			free(store.key);
  			if (store.value_regex != NULL &&
@@@ -2743,8 -2589,8 +2748,8 @@@
  		}
  
  		/* if nothing to unset, or too many matches, error out */
 -		if ((store.seen == 0 && value == NULL) ||
 -				(store.seen > 1 && multi_replace == 0)) {
 +		if ((store.seen_nr == 0 && value == NULL) ||
 +		    (store.seen_nr > 1 && multi_replace == 0)) {
  			ret = CONFIG_NOTHING_SET;
  			goto out_free;
  		}
@@@ -2775,49 -2621,18 +2780,49 @@@
  			goto out_free;
  		}
  
 -		if (store.seen == 0)
 -			store.seen = 1;
 +		if (store.seen_nr == 0) {
 +			if (!store.seen_alloc) {
 +				/* Did not see key nor section */
 +				ALLOC_GROW(store.seen, 1, store.seen_alloc);
 +				store.seen[0] = store.parsed_nr
 +					- !!store.parsed_nr;
 +			}
 +			store.seen_nr = 1;
 +		}
  
 -		for (i = 0, copy_begin = 0; i < store.seen; i++) {
 -			if (store.offset[i] == 0) {
 -				store.offset[i] = copy_end = contents_sz;
 -			} else if (store.state != KEY_SEEN) {
 -				copy_end = store.offset[i];
 -			} else
 -				copy_end = find_beginning_of_line(
 -					contents, contents_sz,
 -					store.offset[i]-2, &new_line);
 +		for (i = 0, copy_begin = 0; i < store.seen_nr; i++) {
 +			size_t replace_end;
 +			int j = store.seen[i];
 +
 +			new_line = 0;
 +			if (!store.key_seen) {
 +				copy_end = store.parsed[j].end;
 +				/* include '\n' when copying section header */
 +				if (copy_end > 0 && copy_end < contents_sz &&
 +				    contents[copy_end - 1] != '\n' &&
 +				    contents[copy_end] == '\n')
 +					copy_end++;
 +				replace_end = copy_end;
 +			} else {
 +				replace_end = store.parsed[j].end;
 +				copy_end = store.parsed[j].begin;
 +				if (!value)
 +					maybe_remove_section(&store, contents,
 +							     &copy_end,
 +							     &replace_end, &i);
 +				/*
 +				 * Swallow preceding white-space on the same
 +				 * line.
 +				 */
 +				while (copy_end > 0 ) {
 +					char c = contents[copy_end - 1];
 +
 +					if (isspace(c) && c != '\n')
 +						copy_end--;
 +					else
 +						break;
 +				}
 +			}
  
  			if (copy_end > 0 && contents[copy_end-1] != '\n')
  				new_line = 1;
@@@ -2831,16 -2646,16 +2836,16 @@@
  				    write_str_in_full(fd, "\n") < 0)
  					goto write_err_out;
  			}
 -			copy_begin = store.offset[i];
 +			copy_begin = replace_end;
  		}
  
  		/* write the pair (value == NULL means unset) */
  		if (value != NULL) {
 -			if (store.state == START) {
 -				if (write_section(fd, key) < 0)
 +			if (!store.section_seen) {
 +				if (write_section(fd, key, &store) < 0)
  					goto write_err_out;
  			}
 -			if (write_pair(fd, key, value) < 0)
 +			if (write_pair(fd, key, value, &store) < 0)
  				goto write_err_out;
  		}
  
@@@ -2964,8 -2779,7 +2969,8 @@@ static int section_name_is_ok(const cha
  
  /* if new_name == NULL, the section is removed instead */
  static int git_config_copy_or_rename_section_in_file(const char *config_filename,
 -				      const char *old_name, const char *new_name, int copy)
 +				      const char *old_name,
 +				      const char *new_name, int copy)
  {
  	int ret = 0, remove = 0;
  	char *filename_buf = NULL;
@@@ -2975,9 -2789,6 +2980,9 @@@
  	FILE *config_file = NULL;
  	struct stat st;
  	struct strbuf copystr = STRBUF_INIT;
 +	struct config_store_data store;
 +
 +	memset(&store, 0, sizeof(store));
  
  	if (new_name && !section_name_is_ok(new_name)) {
  		ret = error("invalid section name: %s", new_name);
@@@ -3047,7 -2858,7 +3052,7 @@@
  				}
  				store.baselen = strlen(new_name);
  				if (!copy) {
 -					if (write_section(out_fd, new_name) < 0) {
 +					if (write_section(out_fd, new_name, &store) < 0) {
  						ret = write_error(get_lock_file_path(&lock));
  						goto out;
  					}
@@@ -3068,7 -2879,7 +3073,7 @@@
  						output[0] = '\t';
  					}
  				} else {
 -					copystr = store_create_section(new_name);
 +					copystr = store_create_section(new_name, &store);
  				}
  			}
  			remove = 0;
diff --combined convert.c
index c480097a2a,1ae6301629..64d0d30e08
--- a/convert.c
+++ b/convert.c
@@@ -7,6 -7,7 +7,7 @@@
  #include "sigchain.h"
  #include "pkt-line.h"
  #include "sub-process.h"
+ #include "utf8.h"
  
  /*
   * convert.c - convert a file when checking it out and checking it in.
@@@ -265,6 -266,241 +266,241 @@@ static int will_convert_lf_to_crlf(size
  
  }
  
+ static int validate_encoding(const char *path, const char *enc,
+ 		      const char *data, size_t len, int die_on_error)
+ {
+ 	/* We only check for UTF here as UTF?? can be an alias for UTF-?? */
+ 	if (istarts_with(enc, "UTF")) {
+ 		/*
+ 		 * Check for detectable errors in UTF encodings
+ 		 */
+ 		if (has_prohibited_utf_bom(enc, data, len)) {
+ 			const char *error_msg = _(
+ 				"BOM is prohibited in '%s' if encoded as %s");
+ 			/*
+ 			 * This advice is shown for UTF-??BE and UTF-??LE encodings.
+ 			 * We cut off the last two characters of the encoding name
+ 			 * to generate the encoding name suitable for BOMs.
+ 			 */
+ 			const char *advise_msg = _(
+ 				"The file '%s' contains a byte order "
+ 				"mark (BOM). Please use UTF-%s as "
+ 				"working-tree-encoding.");
+ 			const char *stripped = NULL;
+ 			char *upper = xstrdup_toupper(enc);
+ 			upper[strlen(upper)-2] = '\0';
+ 			if (!skip_prefix(upper, "UTF-", &stripped))
+ 				skip_prefix(stripped, "UTF", &stripped);
+ 			advise(advise_msg, path, stripped);
+ 			free(upper);
+ 			if (die_on_error)
+ 				die(error_msg, path, enc);
+ 			else {
+ 				return error(error_msg, path, enc);
+ 			}
+ 
+ 		} else if (is_missing_required_utf_bom(enc, data, len)) {
+ 			const char *error_msg = _(
+ 				"BOM is required in '%s' if encoded as %s");
+ 			const char *advise_msg = _(
+ 				"The file '%s' is missing a byte order "
+ 				"mark (BOM). Please use UTF-%sBE or UTF-%sLE "
+ 				"(depending on the byte order) as "
+ 				"working-tree-encoding.");
+ 			const char *stripped = NULL;
+ 			char *upper = xstrdup_toupper(enc);
+ 			if (!skip_prefix(upper, "UTF-", &stripped))
+ 				skip_prefix(stripped, "UTF", &stripped);
+ 			advise(advise_msg, path, stripped, stripped);
+ 			free(upper);
+ 			if (die_on_error)
+ 				die(error_msg, path, enc);
+ 			else {
+ 				return error(error_msg, path, enc);
+ 			}
+ 		}
+ 
+ 	}
+ 	return 0;
+ }
+ 
+ static void trace_encoding(const char *context, const char *path,
+ 			   const char *encoding, const char *buf, size_t len)
+ {
+ 	static struct trace_key coe = TRACE_KEY_INIT(WORKING_TREE_ENCODING);
+ 	struct strbuf trace = STRBUF_INIT;
+ 	int i;
+ 
+ 	strbuf_addf(&trace, "%s (%s, considered %s):\n", context, path, encoding);
+ 	for (i = 0; i < len && buf; ++i) {
+ 		strbuf_addf(
+ 			&trace,"| \e[2m%2i:\e[0m %2x \e[2m%c\e[0m%c",
+ 			i,
+ 			(unsigned char) buf[i],
+ 			(buf[i] > 32 && buf[i] < 127 ? buf[i] : ' '),
+ 			((i+1) % 8 && (i+1) < len ? ' ' : '\n')
+ 		);
+ 	}
+ 	strbuf_addchars(&trace, '\n', 1);
+ 
+ 	trace_strbuf(&coe, &trace);
+ 	strbuf_release(&trace);
+ }
+ 
+ static int check_roundtrip(const char *enc_name)
+ {
+ 	/*
+ 	 * check_roundtrip_encoding contains a string of comma and/or
+ 	 * space separated encodings (eg. "UTF-16, ASCII, CP1125").
+ 	 * Search for the given encoding in that string.
+ 	 */
+ 	const char *found = strcasestr(check_roundtrip_encoding, enc_name);
+ 	const char *next;
+ 	int len;
+ 	if (!found)
+ 		return 0;
+ 	next = found + strlen(enc_name);
+ 	len = strlen(check_roundtrip_encoding);
+ 	return (found && (
+ 			/*
+ 			 * check that the found encoding is at the
+ 			 * beginning of check_roundtrip_encoding or
+ 			 * that it is prefixed with a space or comma
+ 			 */
+ 			found == check_roundtrip_encoding || (
+ 				(isspace(found[-1]) || found[-1] == ',')
+ 			)
+ 		) && (
+ 			/*
+ 			 * check that the found encoding is at the
+ 			 * end of check_roundtrip_encoding or
+ 			 * that it is suffixed with a space or comma
+ 			 */
+ 			next == check_roundtrip_encoding + len || (
+ 				next < check_roundtrip_encoding + len &&
+ 				(isspace(next[0]) || next[0] == ',')
+ 			)
+ 		));
+ }
+ 
+ static const char *default_encoding = "UTF-8";
+ 
+ static int encode_to_git(const char *path, const char *src, size_t src_len,
+ 			 struct strbuf *buf, const char *enc, int conv_flags)
+ {
+ 	char *dst;
+ 	int dst_len;
+ 	int die_on_error = conv_flags & CONV_WRITE_OBJECT;
+ 
+ 	/*
+ 	 * No encoding is specified or there is nothing to encode.
+ 	 * Tell the caller that the content was not modified.
+ 	 */
+ 	if (!enc || (src && !src_len))
+ 		return 0;
+ 
+ 	/*
+ 	 * Looks like we got called from "would_convert_to_git()".
+ 	 * This means Git wants to know if it would encode (= modify!)
+ 	 * the content. Let's answer with "yes", since an encoding was
+ 	 * specified.
+ 	 */
+ 	if (!buf && !src)
+ 		return 1;
+ 
+ 	if (validate_encoding(path, enc, src, src_len, die_on_error))
+ 		return 0;
+ 
+ 	trace_encoding("source", path, enc, src, src_len);
+ 	dst = reencode_string_len(src, src_len, default_encoding, enc,
+ 				  &dst_len);
+ 	if (!dst) {
+ 		/*
+ 		 * We could add the blob "as-is" to Git. However, on checkout
+ 		 * we would try to reencode to the original encoding. This
+ 		 * would fail and we would leave the user with a messed-up
+ 		 * working tree. Let's try to avoid this by screaming loud.
+ 		 */
+ 		const char* msg = _("failed to encode '%s' from %s to %s");
+ 		if (die_on_error)
+ 			die(msg, path, enc, default_encoding);
+ 		else {
+ 			error(msg, path, enc, default_encoding);
+ 			return 0;
+ 		}
+ 	}
+ 	trace_encoding("destination", path, default_encoding, dst, dst_len);
+ 
+ 	/*
+ 	 * UTF supports lossless conversion round tripping [1] and conversions
+ 	 * between UTF and other encodings are mostly round trip safe as
+ 	 * Unicode aims to be a superset of all other character encodings.
+ 	 * However, certain encodings (e.g. SHIFT-JIS) are known to have round
+ 	 * trip issues [2]. Check the round trip conversion for all encodings
+ 	 * listed in core.checkRoundtripEncoding.
+ 	 *
+ 	 * The round trip check is only performed if content is written to Git.
+ 	 * This ensures that no information is lost during conversion to/from
+ 	 * the internal UTF-8 representation.
+ 	 *
+ 	 * Please note, the code below is not tested because I was not able to
+ 	 * generate a faulty round trip without an iconv error. Iconv errors
+ 	 * are already caught above.
+ 	 *
+ 	 * [1] http://unicode.org/faq/utf_bom.html#gen2
+ 	 * [2] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode
+ 	 */
+ 	if (die_on_error && check_roundtrip(enc)) {
+ 		char *re_src;
+ 		int re_src_len;
+ 
+ 		re_src = reencode_string_len(dst, dst_len,
+ 					     enc, default_encoding,
+ 					     &re_src_len);
+ 
+ 		trace_printf("Checking roundtrip encoding for %s...\n", enc);
+ 		trace_encoding("reencoded source", path, enc,
+ 			       re_src, re_src_len);
+ 
+ 		if (!re_src || src_len != re_src_len ||
+ 		    memcmp(src, re_src, src_len)) {
+ 			const char* msg = _("encoding '%s' from %s to %s and "
+ 					    "back is not the same");
+ 			die(msg, path, enc, default_encoding);
+ 		}
+ 
+ 		free(re_src);
+ 	}
+ 
+ 	strbuf_attach(buf, dst, dst_len, dst_len + 1);
+ 	return 1;
+ }
+ 
+ static int encode_to_worktree(const char *path, const char *src, size_t src_len,
+ 			      struct strbuf *buf, const char *enc)
+ {
+ 	char *dst;
+ 	int dst_len;
+ 
+ 	/*
+ 	 * No encoding is specified or there is nothing to encode.
+ 	 * Tell the caller that the content was not modified.
+ 	 */
+ 	if (!enc || (src && !src_len))
+ 		return 0;
+ 
+ 	dst = reencode_string_len(src, src_len, enc, default_encoding,
+ 				  &dst_len);
+ 	if (!dst) {
+ 		error("failed to encode '%s' from %s to %s",
+ 			path, default_encoding, enc);
+ 		return 0;
+ 	}
+ 
+ 	strbuf_attach(buf, dst, dst_len, dst_len + 1);
+ 	return 1;
+ }
+ 
  static int crlf_to_git(const struct index_state *istate,
  		       const char *path, const char *src, size_t len,
  		       struct strbuf *buf,
@@@ -898,7 -1134,7 +1134,7 @@@ static int ident_to_git(const char *pat
  static int ident_to_worktree(const char *path, const char *src, size_t len,
                               struct strbuf *buf, int ident)
  {
 -	unsigned char sha1[20];
 +	struct object_id oid;
  	char *to_free = NULL, *dollar, *spc;
  	int cnt;
  
@@@ -912,9 -1148,9 +1148,9 @@@
  	/* are we "faking" in place editing ? */
  	if (src == buf->buf)
  		to_free = strbuf_detach(buf, NULL);
 -	hash_sha1_file(src, len, "blob", sha1);
 +	hash_object_file(src, len, "blob", &oid);
  
 -	strbuf_grow(buf, len + cnt * 43);
 +	strbuf_grow(buf, len + cnt * (the_hash_algo->hexsz + 3));
  	for (;;) {
  		/* step 1: run to the next '$' */
  		dollar = memchr(src, '$', len);
@@@ -969,7 -1205,7 +1205,7 @@@
  
  		/* step 4: substitute */
  		strbuf_addstr(buf, "Id: ");
 -		strbuf_add(buf, sha1_to_hex(sha1), 40);
 +		strbuf_addstr(buf, oid_to_hex(&oid));
  		strbuf_addstr(buf, " $");
  	}
  	strbuf_add(buf, src, len);
@@@ -978,6 -1214,24 +1214,24 @@@
  	return 1;
  }
  
+ static const char *git_path_check_encoding(struct attr_check_item *check)
+ {
+ 	const char *value = check->value;
+ 
+ 	if (ATTR_UNSET(value) || !strlen(value))
+ 		return NULL;
+ 
+ 	if (ATTR_TRUE(value) || ATTR_FALSE(value)) {
+ 		die(_("true/false are no valid working-tree-encodings"));
+ 	}
+ 
+ 	/* Don't encode to the default encoding */
+ 	if (same_encoding(value, default_encoding))
+ 		return NULL;
+ 
+ 	return value;
+ }
+ 
  static enum crlf_action git_path_check_crlf(struct attr_check_item *check)
  {
  	const char *value = check->value;
@@@ -1033,6 -1287,7 +1287,7 @@@ struct conv_attrs 
  	enum crlf_action attr_action; /* What attr says */
  	enum crlf_action crlf_action; /* When no attr is set, use core.autocrlf */
  	int ident;
+ 	const char *working_tree_encoding; /* Supported encoding or default encoding if NULL */
  };
  
  static void convert_attrs(struct conv_attrs *ca, const char *path)
@@@ -1041,7 -1296,8 +1296,8 @@@
  
  	if (!check) {
  		check = attr_check_initl("crlf", "ident", "filter",
- 					 "eol", "text", NULL);
+ 					 "eol", "text", "working-tree-encoding",
+ 					 NULL);
  		user_convert_tail = &user_convert;
  		git_config(read_convert_config, NULL);
  	}
@@@ -1064,6 -1320,7 +1320,7 @@@
  			else if (eol_attr == EOL_CRLF)
  				ca->crlf_action = CRLF_TEXT_CRLF;
  		}
+ 		ca->working_tree_encoding = git_path_check_encoding(ccheck + 5);
  	} else {
  		ca->drv = NULL;
  		ca->crlf_action = CRLF_UNDEFINED;
@@@ -1144,6 -1401,13 +1401,13 @@@ int convert_to_git(const struct index_s
  		src = dst->buf;
  		len = dst->len;
  	}
+ 
+ 	ret |= encode_to_git(path, src, len, dst, ca.working_tree_encoding, conv_flags);
+ 	if (ret && dst) {
+ 		src = dst->buf;
+ 		len = dst->len;
+ 	}
+ 
  	if (!(conv_flags & CONV_EOL_KEEP_CRLF)) {
  		ret |= crlf_to_git(istate, path, src, len, dst, ca.crlf_action, conv_flags);
  		if (ret && dst) {
@@@ -1167,6 -1431,7 +1431,7 @@@ void convert_to_git_filter_fd(const str
  	if (!apply_filter(path, NULL, 0, fd, dst, ca.drv, CAP_CLEAN, NULL))
  		die("%s: clean filter '%s' failed", path, ca.drv->name);
  
+ 	encode_to_git(path, dst->buf, dst->len, dst, ca.working_tree_encoding, conv_flags);
  	crlf_to_git(istate, path, dst->buf, dst->len, dst, ca.crlf_action, conv_flags);
  	ident_to_git(path, dst->buf, dst->len, dst, ca.ident);
  }
@@@ -1198,6 -1463,12 +1463,12 @@@ static int convert_to_working_tree_inte
  		}
  	}
  
+ 	ret |= encode_to_worktree(path, src, len, dst, ca.working_tree_encoding);
+ 	if (ret) {
+ 		src = dst->buf;
+ 		len = dst->len;
+ 	}
+ 
  	ret_filter = apply_filter(
  		path, src, len, -1, dst, ca.drv, CAP_SMUDGE, dco);
  	if (!ret_filter && ca.drv && ca.drv->required)
@@@ -1510,7 -1781,7 +1781,7 @@@ struct ident_filter 
  	struct stream_filter filter;
  	struct strbuf left;
  	int state;
 -	char ident[45]; /* ": x40 $" */
 +	char ident[GIT_MAX_HEXSZ + 5]; /* ": x40 $" */
  };
  
  static int is_foreign_ident(const char *str)
@@@ -1635,12 -1906,12 +1906,12 @@@ static struct stream_filter_vtbl ident_
  	ident_free_fn,
  };
  
 -static struct stream_filter *ident_filter(const unsigned char *sha1)
 +static struct stream_filter *ident_filter(const struct object_id *oid)
  {
  	struct ident_filter *ident = xmalloc(sizeof(*ident));
  
  	xsnprintf(ident->ident, sizeof(ident->ident),
 -		  ": %s $", sha1_to_hex(sha1));
 +		  ": %s $", oid_to_hex(oid));
  	strbuf_init(&ident->left, 0);
  	ident->filter.vtbl = &ident_vtbl;
  	ident->state = 0;
@@@ -1655,7 -1926,7 +1926,7 @@@
   * Note that you would be crazy to set CRLF, smuge/clean or ident to a
   * large binary blob you would want us not to slurp into the memory!
   */
 -struct stream_filter *get_stream_filter(const char *path, const unsigned char *sha1)
 +struct stream_filter *get_stream_filter(const char *path, const struct object_id *oid)
  {
  	struct conv_attrs ca;
  	struct stream_filter *filter = NULL;
@@@ -1664,11 -1935,14 +1935,14 @@@
  	if (ca.drv && (ca.drv->process || ca.drv->smudge || ca.drv->clean))
  		return NULL;
  
+ 	if (ca.working_tree_encoding)
+ 		return NULL;
+ 
  	if (ca.crlf_action == CRLF_AUTO || ca.crlf_action == CRLF_AUTO_CRLF)
  		return NULL;
  
  	if (ca.ident)
 -		filter = ident_filter(sha1);
 +		filter = ident_filter(oid);
  
  	if (output_eol(ca.crlf_action) == EOL_CRLF)
  		filter = cascade_filter(filter, lf_to_crlf_filter());
diff --combined convert.h
index 2e9b4f49cc,765abfbd60..01385d9288
--- a/convert.h
+++ b/convert.h
@@@ -12,6 -12,7 +12,7 @@@ struct index_state
  #define CONV_EOL_RNDTRP_WARN  (1<<1) /* Warn if CRLF to LF to CRLF is different */
  #define CONV_EOL_RENORMALIZE  (1<<2) /* Convert CRLF to LF */
  #define CONV_EOL_KEEP_CRLF    (1<<3) /* Keep CRLF line endings as is */
+ #define CONV_WRITE_OBJECT     (1<<4) /* Content is written to the index */
  
  extern int global_conv_flags_eol;
  
@@@ -55,6 -56,7 +56,7 @@@ struct delayed_checkout 
  };
  
  extern enum eol core_eol;
+ extern char *check_roundtrip_encoding;
  extern const char *get_cached_convert_stats_ascii(const struct index_state *istate,
  						  const char *path);
  extern const char *get_wt_convert_stats_ascii(const char *path);
@@@ -93,7 -95,7 +95,7 @@@ extern int would_convert_to_git_filter_
  
  struct stream_filter; /* opaque */
  
 -extern struct stream_filter *get_stream_filter(const char *path, const unsigned char *);
 +extern struct stream_filter *get_stream_filter(const char *path, const struct object_id *);
  extern void free_stream_filter(struct stream_filter *);
  extern int is_null_stream_filter(struct stream_filter *);
  
diff --combined environment.c
index f4dd45eb3d,5bae9131ad..2a6de2330b
--- a/environment.c
+++ b/environment.c
@@@ -13,9 -13,6 +13,9 @@@
  #include "refs.h"
  #include "fmt-merge-msg.h"
  #include "commit.h"
 +#include "argv-array.h"
 +#include "object-store.h"
 +#include "chdir-notify.h"
  
  int trust_executable_bit = 1;
  int trust_ctime = 1;
@@@ -30,8 -27,6 +30,8 @@@ int warn_ambiguous_refs = 1
  int warn_on_object_refname_ambiguity = 1;
  int ref_paranoia = -1;
  int repository_format_precious_objects;
 +char *repository_format_partial_clone;
 +const char *core_partial_clone_filter_default;
  const char *git_commit_encoding;
  const char *git_log_output_encoding;
  const char *apply_default_whitespace;
@@@ -51,10 -46,11 +51,11 @@@ const char *editor_program
  const char *askpass_program;
  const char *excludes_file;
  enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;
 -int check_replace_refs = 1;
 +int check_replace_refs = 1; /* NEEDSWORK: rename to read_replace_refs */
  char *git_replace_ref_base;
  enum eol core_eol = EOL_UNSET;
  int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;
+ char *check_roundtrip_encoding = "SHIFT-JIS";
  unsigned whitespace_rule_cfg = WS_DEFAULT_RULE;
  enum branch_track git_branch_track = BRANCH_TRACK_REMOTE;
  enum rebase_setup_type autorebase = AUTOREBASE_NEVER;
@@@ -65,7 -61,6 +66,7 @@@ enum push_default_type push_default = P
  enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;
  char *notes_ref_name;
  int grafts_replace_parents = 1;
 +int core_commit_graph;
  int core_apply_sparse_checkout;
  int merge_log_config = -1;
  int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
@@@ -104,7 -99,7 +105,7 @@@ int ignore_untracked_cache_config
  /* This is set by setup_git_dir_gently() and/or git_default_config() */
  char *git_work_tree_cfg;
  
 -static char *namespace;
 +static char *git_namespace;
  
  static const char *super_prefix;
  
@@@ -151,35 -146,10 +152,35 @@@ static char *expand_namespace(const cha
  	return strbuf_detach(&buf, NULL);
  }
  
 -void setup_git_env(void)
 +/*
 + * Wrapper of getenv() that returns a strdup value. This value is kept
 + * in argv to be freed later.
 + */
 +static const char *getenv_safe(struct argv_array *argv, const char *name)
 +{
 +	const char *value = getenv(name);
 +
 +	if (!value)
 +		return NULL;
 +
 +	argv_array_push(argv, value);
 +	return argv->argv[argv->argc - 1];
 +}
 +
 +void setup_git_env(const char *git_dir)
  {
  	const char *shallow_file;
  	const char *replace_ref_base;
 +	struct set_gitdir_args args = { NULL };
 +	struct argv_array to_free = ARGV_ARRAY_INIT;
 +
 +	args.commondir = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
 +	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
 +	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
 +	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
 +	args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
 +	repo_set_gitdir(the_repository, git_dir, &args);
 +	argv_array_clear(&to_free);
  
  	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
  		check_replace_refs = 0;
@@@ -187,8 -157,8 +188,8 @@@
  	free(git_replace_ref_base);
  	git_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base
  							  : "refs/replace/");
 -	free(namespace);
 -	namespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));
 +	free(git_namespace);
 +	git_namespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));
  	shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
  	if (shallow_file)
  		set_alternate_shallow_file(shallow_file, 0);
@@@ -222,9 -192,9 +223,9 @@@ const char *get_git_common_dir(void
  
  const char *get_git_namespace(void)
  {
 -	if (!namespace)
 +	if (!git_namespace)
  		BUG("git environment hasn't been setup");
 -	return namespace;
 +	return git_namespace;
  }
  
  const char *strip_namespace(const char *namespaced_ref)
@@@ -273,12 -243,12 +274,12 @@@ const char *get_git_work_tree(void
  
  char *get_object_directory(void)
  {
 -	if (!the_repository->objectdir)
 +	if (!the_repository->objects->objectdir)
  		BUG("git environment hasn't been setup");
 -	return the_repository->objectdir;
 +	return the_repository->objects->objectdir;
  }
  
 -int odb_mkstemp(struct strbuf *template, const char *pattern)
 +int odb_mkstemp(struct strbuf *temp_filename, const char *pattern)
  {
  	int fd;
  	/*
@@@ -286,16 -256,16 +287,16 @@@
  	 * restrictive except to remove write permission.
  	 */
  	int mode = 0444;
 -	git_path_buf(template, "objects/%s", pattern);
 -	fd = git_mkstemp_mode(template->buf, mode);
 +	git_path_buf(temp_filename, "objects/%s", pattern);
 +	fd = git_mkstemp_mode(temp_filename->buf, mode);
  	if (0 <= fd)
  		return fd;
  
  	/* slow path */
 -	/* some mkstemp implementations erase template on failure */
 -	git_path_buf(template, "objects/%s", pattern);
 -	safe_create_leading_directories(template->buf);
 -	return xmkstemp_mode(template->buf, mode);
 +	/* some mkstemp implementations erase temp_filename on failure */
 +	git_path_buf(temp_filename, "objects/%s", pattern);
 +	safe_create_leading_directories(temp_filename->buf);
 +	return xmkstemp_mode(temp_filename->buf, mode);
  }
  
  int odb_pack_keep(const char *name)
@@@ -325,31 -295,13 +326,31 @@@ char *get_graft_file(void
  	return the_repository->graft_file;
  }
  
 -int set_git_dir(const char *path)
 +static void set_git_dir_1(const char *path)
  {
  	if (setenv(GIT_DIR_ENVIRONMENT, path, 1))
 -		return error("Could not set GIT_DIR to '%s'", path);
 -	repo_set_gitdir(the_repository, path);
 -	setup_git_env();
 -	return 0;
 +		die("could not set GIT_DIR to '%s'", path);
 +	setup_git_env(path);
 +}
 +
 +static void update_relative_gitdir(const char *name,
 +				   const char *old_cwd,
 +				   const char *new_cwd,
 +				   void *data)
 +{
 +	char *path = reparent_relative_path(old_cwd, new_cwd, get_git_dir());
 +	trace_printf_key(&trace_setup_key,
 +			 "setup: move $GIT_DIR to '%s'",
 +			 path);
 +	set_git_dir_1(path);
 +	free(path);
 +}
 +
 +void set_git_dir(const char *path)
 +{
 +	set_git_dir_1(path);
 +	if (!is_absolute_path(path))
 +		chdir_notify_register(NULL, update_relative_gitdir, NULL);
  }
  
  const char *get_log_output_encoding(void)
diff --combined git-compat-util.h
index 07e383257b,95c9b34832..f1d7ada890
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@@ -455,6 -455,7 +455,7 @@@ extern void (*get_warn_routine(void))(c
  extern void set_die_is_recursing_routine(int (*routine)(void));
  
  extern int starts_with(const char *str, const char *prefix);
+ extern int istarts_with(const char *str, const char *prefix);
  
  /*
   * If the string "str" begins with the string found in "prefix", return 1.
@@@ -826,8 -827,8 +827,8 @@@ extern ssize_t xpread(int fd, void *buf
  extern int xdup(int fd);
  extern FILE *xfopen(const char *path, const char *mode);
  extern FILE *xfdopen(int fd, const char *mode);
 -extern int xmkstemp(char *template);
 -extern int xmkstemp_mode(char *template, int mode);
 +extern int xmkstemp(char *temp_filename);
 +extern int xmkstemp_mode(char *temp_filename, int mode);
  extern char *xgetcwd(void);
  extern FILE *fopen_for_writing(const char *path);
  extern FILE *fopen_or_warn(const char *path, const char *mode);
diff --combined sha1-file.c
index 64a5bd7d87,0000000000..46072602ff
mode 100644,000000..100644
--- a/sha1-file.c
+++ b/sha1-file.c
@@@ -1,2255 -1,0 +1,2255 @@@
 +/*
 + * GIT - The information manager from hell
 + *
 + * Copyright (C) Linus Torvalds, 2005
 + *
 + * This handles basic git sha1 object files - packing, unpacking,
 + * creation etc.
 + */
 +#include "cache.h"
 +#include "config.h"
 +#include "string-list.h"
 +#include "lockfile.h"
 +#include "delta.h"
 +#include "pack.h"
 +#include "blob.h"
 +#include "commit.h"
 +#include "run-command.h"
 +#include "tag.h"
 +#include "tree.h"
 +#include "tree-walk.h"
 +#include "refs.h"
 +#include "pack-revindex.h"
 +#include "sha1-lookup.h"
 +#include "bulk-checkin.h"
 +#include "repository.h"
 +#include "replace-object.h"
 +#include "streaming.h"
 +#include "dir.h"
 +#include "list.h"
 +#include "mergesort.h"
 +#include "quote.h"
 +#include "packfile.h"
 +#include "fetch-object.h"
 +#include "object-store.h"
 +
 +/* The maximum size for an object header. */
 +#define MAX_HEADER_LEN 32
 +
 +const unsigned char null_sha1[GIT_MAX_RAWSZ];
 +const struct object_id null_oid;
 +const struct object_id empty_tree_oid = {
 +	EMPTY_TREE_SHA1_BIN_LITERAL
 +};
 +const struct object_id empty_blob_oid = {
 +	EMPTY_BLOB_SHA1_BIN_LITERAL
 +};
 +
 +static void git_hash_sha1_init(git_hash_ctx *ctx)
 +{
 +	git_SHA1_Init(&ctx->sha1);
 +}
 +
 +static void git_hash_sha1_update(git_hash_ctx *ctx, const void *data, size_t len)
 +{
 +	git_SHA1_Update(&ctx->sha1, data, len);
 +}
 +
 +static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)
 +{
 +	git_SHA1_Final(hash, &ctx->sha1);
 +}
 +
 +static void git_hash_unknown_init(git_hash_ctx *ctx)
 +{
 +	die("trying to init unknown hash");
 +}
 +
 +static void git_hash_unknown_update(git_hash_ctx *ctx, const void *data, size_t len)
 +{
 +	die("trying to update unknown hash");
 +}
 +
 +static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)
 +{
 +	die("trying to finalize unknown hash");
 +}
 +
 +const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 +	{
 +		NULL,
 +		0x00000000,
 +		0,
 +		0,
 +		git_hash_unknown_init,
 +		git_hash_unknown_update,
 +		git_hash_unknown_final,
 +		NULL,
 +		NULL,
 +	},
 +	{
 +		"sha-1",
 +		/* "sha1", big-endian */
 +		0x73686131,
 +		GIT_SHA1_RAWSZ,
 +		GIT_SHA1_HEXSZ,
 +		git_hash_sha1_init,
 +		git_hash_sha1_update,
 +		git_hash_sha1_final,
 +		&empty_tree_oid,
 +		&empty_blob_oid,
 +	},
 +};
 +
 +/*
 + * This is meant to hold a *small* number of objects that you would
 + * want read_sha1_file() to be able to return, but yet you do not want
 + * to write them into the object store (e.g. a browse-only
 + * application).
 + */
 +static struct cached_object {
 +	unsigned char sha1[20];
 +	enum object_type type;
 +	void *buf;
 +	unsigned long size;
 +} *cached_objects;
 +static int cached_object_nr, cached_object_alloc;
 +
 +static struct cached_object empty_tree = {
 +	EMPTY_TREE_SHA1_BIN_LITERAL,
 +	OBJ_TREE,
 +	"",
 +	0
 +};
 +
 +static struct cached_object *find_cached_object(const unsigned char *sha1)
 +{
 +	int i;
 +	struct cached_object *co = cached_objects;
 +
 +	for (i = 0; i < cached_object_nr; i++, co++) {
 +		if (!hashcmp(co->sha1, sha1))
 +			return co;
 +	}
 +	if (!hashcmp(sha1, empty_tree.sha1))
 +		return &empty_tree;
 +	return NULL;
 +}
 +
 +
 +static int get_conv_flags(unsigned flags)
 +{
 +	if (flags & HASH_RENORMALIZE)
 +		return CONV_EOL_RENORMALIZE;
 +	else if (flags & HASH_WRITE_OBJECT)
- 	  return global_conv_flags_eol;
++		return global_conv_flags_eol | CONV_WRITE_OBJECT;
 +	else
 +		return 0;
 +}
 +
 +
 +int mkdir_in_gitdir(const char *path)
 +{
 +	if (mkdir(path, 0777)) {
 +		int saved_errno = errno;
 +		struct stat st;
 +		struct strbuf sb = STRBUF_INIT;
 +
 +		if (errno != EEXIST)
 +			return -1;
 +		/*
 +		 * Are we looking at a path in a symlinked worktree
 +		 * whose original repository does not yet have it?
 +		 * e.g. .git/rr-cache pointing at its original
 +		 * repository in which the user hasn't performed any
 +		 * conflict resolution yet?
 +		 */
 +		if (lstat(path, &st) || !S_ISLNK(st.st_mode) ||
 +		    strbuf_readlink(&sb, path, st.st_size) ||
 +		    !is_absolute_path(sb.buf) ||
 +		    mkdir(sb.buf, 0777)) {
 +			strbuf_release(&sb);
 +			errno = saved_errno;
 +			return -1;
 +		}
 +		strbuf_release(&sb);
 +	}
 +	return adjust_shared_perm(path);
 +}
 +
 +enum scld_error safe_create_leading_directories(char *path)
 +{
 +	char *next_component = path + offset_1st_component(path);
 +	enum scld_error ret = SCLD_OK;
 +
 +	while (ret == SCLD_OK && next_component) {
 +		struct stat st;
 +		char *slash = next_component, slash_character;
 +
 +		while (*slash && !is_dir_sep(*slash))
 +			slash++;
 +
 +		if (!*slash)
 +			break;
 +
 +		next_component = slash + 1;
 +		while (is_dir_sep(*next_component))
 +			next_component++;
 +		if (!*next_component)
 +			break;
 +
 +		slash_character = *slash;
 +		*slash = '\0';
 +		if (!stat(path, &st)) {
 +			/* path exists */
 +			if (!S_ISDIR(st.st_mode)) {
 +				errno = ENOTDIR;
 +				ret = SCLD_EXISTS;
 +			}
 +		} else if (mkdir(path, 0777)) {
 +			if (errno == EEXIST &&
 +			    !stat(path, &st) && S_ISDIR(st.st_mode))
 +				; /* somebody created it since we checked */
 +			else if (errno == ENOENT)
 +				/*
 +				 * Either mkdir() failed because
 +				 * somebody just pruned the containing
 +				 * directory, or stat() failed because
 +				 * the file that was in our way was
 +				 * just removed.  Either way, inform
 +				 * the caller that it might be worth
 +				 * trying again:
 +				 */
 +				ret = SCLD_VANISHED;
 +			else
 +				ret = SCLD_FAILED;
 +		} else if (adjust_shared_perm(path)) {
 +			ret = SCLD_PERMS;
 +		}
 +		*slash = slash_character;
 +	}
 +	return ret;
 +}
 +
 +enum scld_error safe_create_leading_directories_const(const char *path)
 +{
 +	int save_errno;
 +	/* path points to cache entries, so xstrdup before messing with it */
 +	char *buf = xstrdup(path);
 +	enum scld_error result = safe_create_leading_directories(buf);
 +
 +	save_errno = errno;
 +	free(buf);
 +	errno = save_errno;
 +	return result;
 +}
 +
 +int raceproof_create_file(const char *path, create_file_fn fn, void *cb)
 +{
 +	/*
 +	 * The number of times we will try to remove empty directories
 +	 * in the way of path. This is only 1 because if another
 +	 * process is racily creating directories that conflict with
 +	 * us, we don't want to fight against them.
 +	 */
 +	int remove_directories_remaining = 1;
 +
 +	/*
 +	 * The number of times that we will try to create the
 +	 * directories containing path. We are willing to attempt this
 +	 * more than once, because another process could be trying to
 +	 * clean up empty directories at the same time as we are
 +	 * trying to create them.
 +	 */
 +	int create_directories_remaining = 3;
 +
 +	/* A scratch copy of path, filled lazily if we need it: */
 +	struct strbuf path_copy = STRBUF_INIT;
 +
 +	int ret, save_errno;
 +
 +	/* Sanity check: */
 +	assert(*path);
 +
 +retry_fn:
 +	ret = fn(path, cb);
 +	save_errno = errno;
 +	if (!ret)
 +		goto out;
 +
 +	if (errno == EISDIR && remove_directories_remaining-- > 0) {
 +		/*
 +		 * A directory is in the way. Maybe it is empty; try
 +		 * to remove it:
 +		 */
 +		if (!path_copy.len)
 +			strbuf_addstr(&path_copy, path);
 +
 +		if (!remove_dir_recursively(&path_copy, REMOVE_DIR_EMPTY_ONLY))
 +			goto retry_fn;
 +	} else if (errno == ENOENT && create_directories_remaining-- > 0) {
 +		/*
 +		 * Maybe the containing directory didn't exist, or
 +		 * maybe it was just deleted by a process that is
 +		 * racing with us to clean up empty directories. Try
 +		 * to create it:
 +		 */
 +		enum scld_error scld_result;
 +
 +		if (!path_copy.len)
 +			strbuf_addstr(&path_copy, path);
 +
 +		do {
 +			scld_result = safe_create_leading_directories(path_copy.buf);
 +			if (scld_result == SCLD_OK)
 +				goto retry_fn;
 +		} while (scld_result == SCLD_VANISHED && create_directories_remaining-- > 0);
 +	}
 +
 +out:
 +	strbuf_release(&path_copy);
 +	errno = save_errno;
 +	return ret;
 +}
 +
 +static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 +{
 +	int i;
 +	for (i = 0; i < 20; i++) {
 +		static char hex[] = "0123456789abcdef";
 +		unsigned int val = sha1[i];
 +		strbuf_addch(buf, hex[val >> 4]);
 +		strbuf_addch(buf, hex[val & 0xf]);
 +		if (!i)
 +			strbuf_addch(buf, '/');
 +	}
 +}
 +
 +void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
 +{
 +	strbuf_addstr(buf, r->objects->objectdir);
 +	strbuf_addch(buf, '/');
 +	fill_sha1_path(buf, sha1);
 +}
 +
 +struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)
 +{
 +	strbuf_setlen(&alt->scratch, alt->base_len);
 +	return &alt->scratch;
 +}
 +
 +static const char *alt_sha1_path(struct alternate_object_database *alt,
 +				 const unsigned char *sha1)
 +{
 +	struct strbuf *buf = alt_scratch_buf(alt);
 +	fill_sha1_path(buf, sha1);
 +	return buf->buf;
 +}
 +
 +/*
 + * Return non-zero iff the path is usable as an alternate object database.
 + */
 +static int alt_odb_usable(struct raw_object_store *o,
 +			  struct strbuf *path,
 +			  const char *normalized_objdir)
 +{
 +	struct alternate_object_database *alt;
 +
 +	/* Detect cases where alternate disappeared */
 +	if (!is_directory(path->buf)) {
 +		error("object directory %s does not exist; "
 +		      "check .git/objects/info/alternates.",
 +		      path->buf);
 +		return 0;
 +	}
 +
 +	/*
 +	 * Prevent the common mistake of listing the same
 +	 * thing twice, or object directory itself.
 +	 */
 +	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 +		if (!fspathcmp(path->buf, alt->path))
 +			return 0;
 +	}
 +	if (!fspathcmp(path->buf, normalized_objdir))
 +		return 0;
 +
 +	return 1;
 +}
 +
 +/*
 + * Prepare alternate object database registry.
 + *
 + * The variable alt_odb_list points at the list of struct
 + * alternate_object_database.  The elements on this list come from
 + * non-empty elements from colon separated ALTERNATE_DB_ENVIRONMENT
 + * environment variable, and $GIT_OBJECT_DIRECTORY/info/alternates,
 + * whose contents is similar to that environment variable but can be
 + * LF separated.  Its base points at a statically allocated buffer that
 + * contains "/the/directory/corresponding/to/.git/objects/...", while
 + * its name points just after the slash at the end of ".git/objects/"
 + * in the example above, and has enough space to hold 40-byte hex
 + * SHA1, an extra slash for the first level indirection, and the
 + * terminating NUL.
 + */
 +static void read_info_alternates(struct repository *r,
 +				 const char *relative_base,
 +				 int depth);
 +static int link_alt_odb_entry(struct repository *r, const char *entry,
 +	const char *relative_base, int depth, const char *normalized_objdir)
 +{
 +	struct alternate_object_database *ent;
 +	struct strbuf pathbuf = STRBUF_INIT;
 +
 +	if (!is_absolute_path(entry) && relative_base) {
 +		strbuf_realpath(&pathbuf, relative_base, 1);
 +		strbuf_addch(&pathbuf, '/');
 +	}
 +	strbuf_addstr(&pathbuf, entry);
 +
 +	if (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {
 +		error("unable to normalize alternate object path: %s",
 +		      pathbuf.buf);
 +		strbuf_release(&pathbuf);
 +		return -1;
 +	}
 +
 +	/*
 +	 * The trailing slash after the directory name is given by
 +	 * this function at the end. Remove duplicates.
 +	 */
 +	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 +		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 +
 +	if (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {
 +		strbuf_release(&pathbuf);
 +		return -1;
 +	}
 +
 +	ent = alloc_alt_odb(pathbuf.buf);
 +
 +	/* add the alternate entry */
 +	*r->objects->alt_odb_tail = ent;
 +	r->objects->alt_odb_tail = &(ent->next);
 +	ent->next = NULL;
 +
 +	/* recursively add alternates */
 +	read_info_alternates(r, pathbuf.buf, depth + 1);
 +
 +	strbuf_release(&pathbuf);
 +	return 0;
 +}
 +
 +static const char *parse_alt_odb_entry(const char *string,
 +				       int sep,
 +				       struct strbuf *out)
 +{
 +	const char *end;
 +
 +	strbuf_reset(out);
 +
 +	if (*string == '#') {
 +		/* comment; consume up to next separator */
 +		end = strchrnul(string, sep);
 +	} else if (*string == '"' && !unquote_c_style(out, string, &end)) {
 +		/*
 +		 * quoted path; unquote_c_style has copied the
 +		 * data for us and set "end". Broken quoting (e.g.,
 +		 * an entry that doesn't end with a quote) falls
 +		 * back to the unquoted case below.
 +		 */
 +	} else {
 +		/* normal, unquoted path */
 +		end = strchrnul(string, sep);
 +		strbuf_add(out, string, end - string);
 +	}
 +
 +	if (*end)
 +		end++;
 +	return end;
 +}
 +
 +static void link_alt_odb_entries(struct repository *r, const char *alt,
 +				 int sep, const char *relative_base, int depth)
 +{
 +	struct strbuf objdirbuf = STRBUF_INIT;
 +	struct strbuf entry = STRBUF_INIT;
 +
 +	if (!alt || !*alt)
 +		return;
 +
 +	if (depth > 5) {
 +		error("%s: ignoring alternate object stores, nesting too deep.",
 +				relative_base);
 +		return;
 +	}
 +
 +	strbuf_add_absolute_path(&objdirbuf, r->objects->objectdir);
 +	if (strbuf_normalize_path(&objdirbuf) < 0)
 +		die("unable to normalize object directory: %s",
 +		    objdirbuf.buf);
 +
 +	while (*alt) {
 +		alt = parse_alt_odb_entry(alt, sep, &entry);
 +		if (!entry.len)
 +			continue;
 +		link_alt_odb_entry(r, entry.buf,
 +				   relative_base, depth, objdirbuf.buf);
 +	}
 +	strbuf_release(&entry);
 +	strbuf_release(&objdirbuf);
 +}
 +
 +static void read_info_alternates(struct repository *r,
 +				 const char *relative_base,
 +				 int depth)
 +{
 +	char *path;
 +	struct strbuf buf = STRBUF_INIT;
 +
 +	path = xstrfmt("%s/info/alternates", relative_base);
 +	if (strbuf_read_file(&buf, path, 1024) < 0) {
 +		warn_on_fopen_errors(path);
 +		free(path);
 +		return;
 +	}
 +
 +	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 +	strbuf_release(&buf);
 +	free(path);
 +}
 +
 +struct alternate_object_database *alloc_alt_odb(const char *dir)
 +{
 +	struct alternate_object_database *ent;
 +
 +	FLEX_ALLOC_STR(ent, path, dir);
 +	strbuf_init(&ent->scratch, 0);
 +	strbuf_addf(&ent->scratch, "%s/", dir);
 +	ent->base_len = ent->scratch.len;
 +
 +	return ent;
 +}
 +
 +void add_to_alternates_file(const char *reference)
 +{
 +	struct lock_file lock = LOCK_INIT;
 +	char *alts = git_pathdup("objects/info/alternates");
 +	FILE *in, *out;
 +	int found = 0;
 +
 +	hold_lock_file_for_update(&lock, alts, LOCK_DIE_ON_ERROR);
 +	out = fdopen_lock_file(&lock, "w");
 +	if (!out)
 +		die_errno("unable to fdopen alternates lockfile");
 +
 +	in = fopen(alts, "r");
 +	if (in) {
 +		struct strbuf line = STRBUF_INIT;
 +
 +		while (strbuf_getline(&line, in) != EOF) {
 +			if (!strcmp(reference, line.buf)) {
 +				found = 1;
 +				break;
 +			}
 +			fprintf_or_die(out, "%s\n", line.buf);
 +		}
 +
 +		strbuf_release(&line);
 +		fclose(in);
 +	}
 +	else if (errno != ENOENT)
 +		die_errno("unable to read alternates file");
 +
 +	if (found) {
 +		rollback_lock_file(&lock);
 +	} else {
 +		fprintf_or_die(out, "%s\n", reference);
 +		if (commit_lock_file(&lock))
 +			die_errno("unable to move new alternates file into place");
 +		if (the_repository->objects->alt_odb_tail)
 +			link_alt_odb_entries(the_repository, reference,
 +					     '\n', NULL, 0);
 +	}
 +	free(alts);
 +}
 +
 +void add_to_alternates_memory(const char *reference)
 +{
 +	/*
 +	 * Make sure alternates are initialized, or else our entry may be
 +	 * overwritten when they are.
 +	 */
 +	prepare_alt_odb(the_repository);
 +
 +	link_alt_odb_entries(the_repository, reference,
 +			     '\n', NULL, 0);
 +}
 +
 +/*
 + * Compute the exact path an alternate is at and returns it. In case of
 + * error NULL is returned and the human readable error is added to `err`
 + * `path` may be relative and should point to $GITDIR.
 + * `err` must not be null.
 + */
 +char *compute_alternate_path(const char *path, struct strbuf *err)
 +{
 +	char *ref_git = NULL;
 +	const char *repo, *ref_git_s;
 +	int seen_error = 0;
 +
 +	ref_git_s = real_path_if_valid(path);
 +	if (!ref_git_s) {
 +		seen_error = 1;
 +		strbuf_addf(err, _("path '%s' does not exist"), path);
 +		goto out;
 +	} else
 +		/*
 +		 * Beware: read_gitfile(), real_path() and mkpath()
 +		 * return static buffer
 +		 */
 +		ref_git = xstrdup(ref_git_s);
 +
 +	repo = read_gitfile(ref_git);
 +	if (!repo)
 +		repo = read_gitfile(mkpath("%s/.git", ref_git));
 +	if (repo) {
 +		free(ref_git);
 +		ref_git = xstrdup(repo);
 +	}
 +
 +	if (!repo && is_directory(mkpath("%s/.git/objects", ref_git))) {
 +		char *ref_git_git = mkpathdup("%s/.git", ref_git);
 +		free(ref_git);
 +		ref_git = ref_git_git;
 +	} else if (!is_directory(mkpath("%s/objects", ref_git))) {
 +		struct strbuf sb = STRBUF_INIT;
 +		seen_error = 1;
 +		if (get_common_dir(&sb, ref_git)) {
 +			strbuf_addf(err,
 +				    _("reference repository '%s' as a linked "
 +				      "checkout is not supported yet."),
 +				    path);
 +			goto out;
 +		}
 +
 +		strbuf_addf(err, _("reference repository '%s' is not a "
 +					"local repository."), path);
 +		goto out;
 +	}
 +
 +	if (!access(mkpath("%s/shallow", ref_git), F_OK)) {
 +		strbuf_addf(err, _("reference repository '%s' is shallow"),
 +			    path);
 +		seen_error = 1;
 +		goto out;
 +	}
 +
 +	if (!access(mkpath("%s/info/grafts", ref_git), F_OK)) {
 +		strbuf_addf(err,
 +			    _("reference repository '%s' is grafted"),
 +			    path);
 +		seen_error = 1;
 +		goto out;
 +	}
 +
 +out:
 +	if (seen_error) {
 +		FREE_AND_NULL(ref_git);
 +	}
 +
 +	return ref_git;
 +}
 +
 +int foreach_alt_odb(alt_odb_fn fn, void *cb)
 +{
 +	struct alternate_object_database *ent;
 +	int r = 0;
 +
 +	prepare_alt_odb(the_repository);
 +	for (ent = the_repository->objects->alt_odb_list; ent; ent = ent->next) {
 +		r = fn(ent, cb);
 +		if (r)
 +			break;
 +	}
 +	return r;
 +}
 +
 +void prepare_alt_odb(struct repository *r)
 +{
 +	if (r->objects->alt_odb_tail)
 +		return;
 +
 +	r->objects->alt_odb_tail = &r->objects->alt_odb_list;
 +	link_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);
 +
 +	read_info_alternates(r, r->objects->objectdir, 0);
 +}
 +
 +/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
 +static int freshen_file(const char *fn)
 +{
 +	struct utimbuf t;
 +	t.actime = t.modtime = time(NULL);
 +	return !utime(fn, &t);
 +}
 +
 +/*
 + * All of the check_and_freshen functions return 1 if the file exists and was
 + * freshened (if freshening was requested), 0 otherwise. If they return
 + * 0, you should not assume that it is safe to skip a write of the object (it
 + * either does not exist on disk, or has a stale mtime and may be subject to
 + * pruning).
 + */
 +int check_and_freshen_file(const char *fn, int freshen)
 +{
 +	if (access(fn, F_OK))
 +		return 0;
 +	if (freshen && !freshen_file(fn))
 +		return 0;
 +	return 1;
 +}
 +
 +static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 +{
 +	static struct strbuf buf = STRBUF_INIT;
 +
 +	strbuf_reset(&buf);
 +	sha1_file_name(the_repository, &buf, sha1);
 +
 +	return check_and_freshen_file(buf.buf, freshen);
 +}
 +
 +static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 +{
 +	struct alternate_object_database *alt;
 +	prepare_alt_odb(the_repository);
 +	for (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {
 +		const char *path = alt_sha1_path(alt, sha1);
 +		if (check_and_freshen_file(path, freshen))
 +			return 1;
 +	}
 +	return 0;
 +}
 +
 +static int check_and_freshen(const unsigned char *sha1, int freshen)
 +{
 +	return check_and_freshen_local(sha1, freshen) ||
 +	       check_and_freshen_nonlocal(sha1, freshen);
 +}
 +
 +int has_loose_object_nonlocal(const unsigned char *sha1)
 +{
 +	return check_and_freshen_nonlocal(sha1, 0);
 +}
 +
 +static int has_loose_object(const unsigned char *sha1)
 +{
 +	return check_and_freshen(sha1, 0);
 +}
 +
 +static void mmap_limit_check(size_t length)
 +{
 +	static size_t limit = 0;
 +	if (!limit) {
 +		limit = git_env_ulong("GIT_MMAP_LIMIT", 0);
 +		if (!limit)
 +			limit = SIZE_MAX;
 +	}
 +	if (length > limit)
 +		die("attempting to mmap %"PRIuMAX" over limit %"PRIuMAX,
 +		    (uintmax_t)length, (uintmax_t)limit);
 +}
 +
 +void *xmmap_gently(void *start, size_t length,
 +		  int prot, int flags, int fd, off_t offset)
 +{
 +	void *ret;
 +
 +	mmap_limit_check(length);
 +	ret = mmap(start, length, prot, flags, fd, offset);
 +	if (ret == MAP_FAILED) {
 +		if (!length)
 +			return NULL;
 +		release_pack_memory(length);
 +		ret = mmap(start, length, prot, flags, fd, offset);
 +	}
 +	return ret;
 +}
 +
 +void *xmmap(void *start, size_t length,
 +	int prot, int flags, int fd, off_t offset)
 +{
 +	void *ret = xmmap_gently(start, length, prot, flags, fd, offset);
 +	if (ret == MAP_FAILED)
 +		die_errno("mmap failed");
 +	return ret;
 +}
 +
 +/*
 + * With an in-core object data in "map", rehash it to make sure the
 + * object name actually matches "sha1" to detect object corruption.
 + * With "map" == NULL, try reading the object named with "sha1" using
 + * the streaming interface and rehash it to do the same.
 + */
 +int check_object_signature(const struct object_id *oid, void *map,
 +			   unsigned long size, const char *type)
 +{
 +	struct object_id real_oid;
 +	enum object_type obj_type;
 +	struct git_istream *st;
 +	git_hash_ctx c;
 +	char hdr[MAX_HEADER_LEN];
 +	int hdrlen;
 +
 +	if (map) {
 +		hash_object_file(map, size, type, &real_oid);
 +		return oidcmp(oid, &real_oid) ? -1 : 0;
 +	}
 +
 +	st = open_istream(oid, &obj_type, &size, NULL);
 +	if (!st)
 +		return -1;
 +
 +	/* Generate the header */
 +	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", type_name(obj_type), size) + 1;
 +
 +	/* Sha1.. */
 +	the_hash_algo->init_fn(&c);
 +	the_hash_algo->update_fn(&c, hdr, hdrlen);
 +	for (;;) {
 +		char buf[1024 * 16];
 +		ssize_t readlen = read_istream(st, buf, sizeof(buf));
 +
 +		if (readlen < 0) {
 +			close_istream(st);
 +			return -1;
 +		}
 +		if (!readlen)
 +			break;
 +		the_hash_algo->update_fn(&c, buf, readlen);
 +	}
 +	the_hash_algo->final_fn(real_oid.hash, &c);
 +	close_istream(st);
 +	return oidcmp(oid, &real_oid) ? -1 : 0;
 +}
 +
 +int git_open_cloexec(const char *name, int flags)
 +{
 +	int fd;
 +	static int o_cloexec = O_CLOEXEC;
 +
 +	fd = open(name, flags | o_cloexec);
 +	if ((o_cloexec & O_CLOEXEC) && fd < 0 && errno == EINVAL) {
 +		/* Try again w/o O_CLOEXEC: the kernel might not support it */
 +		o_cloexec &= ~O_CLOEXEC;
 +		fd = open(name, flags | o_cloexec);
 +	}
 +
 +#if defined(F_GETFD) && defined(F_SETFD) && defined(FD_CLOEXEC)
 +	{
 +		static int fd_cloexec = FD_CLOEXEC;
 +
 +		if (!o_cloexec && 0 <= fd && fd_cloexec) {
 +			/* Opened w/o O_CLOEXEC?  try with fcntl(2) to add it */
 +			int flags = fcntl(fd, F_GETFD);
 +			if (fcntl(fd, F_SETFD, flags | fd_cloexec))
 +				fd_cloexec = 0;
 +		}
 +	}
 +#endif
 +	return fd;
 +}
 +
 +/*
 + * Find "sha1" as a loose object in the local repository or in an alternate.
 + * Returns 0 on success, negative on failure.
 + *
 + * The "path" out-parameter will give the path of the object we found (if any).
 + * Note that it may point to static storage and is only valid until another
 + * call to sha1_file_name(), etc.
 + */
 +static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
 +			  struct stat *st, const char **path)
 +{
 +	struct alternate_object_database *alt;
 +	static struct strbuf buf = STRBUF_INIT;
 +
 +	strbuf_reset(&buf);
 +	sha1_file_name(r, &buf, sha1);
 +	*path = buf.buf;
 +
 +	if (!lstat(*path, st))
 +		return 0;
 +
 +	prepare_alt_odb(r);
 +	errno = ENOENT;
 +	for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
 +		*path = alt_sha1_path(alt, sha1);
 +		if (!lstat(*path, st))
 +			return 0;
 +	}
 +
 +	return -1;
 +}
 +
 +/*
 + * Like stat_sha1_file(), but actually open the object and return the
 + * descriptor. See the caveats on the "path" parameter above.
 + */
 +static int open_sha1_file(struct repository *r,
 +			  const unsigned char *sha1, const char **path)
 +{
 +	int fd;
 +	struct alternate_object_database *alt;
 +	int most_interesting_errno;
 +	static struct strbuf buf = STRBUF_INIT;
 +
 +	strbuf_reset(&buf);
 +	sha1_file_name(r, &buf, sha1);
 +	*path = buf.buf;
 +
 +	fd = git_open(*path);
 +	if (fd >= 0)
 +		return fd;
 +	most_interesting_errno = errno;
 +
 +	prepare_alt_odb(r);
 +	for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
 +		*path = alt_sha1_path(alt, sha1);
 +		fd = git_open(*path);
 +		if (fd >= 0)
 +			return fd;
 +		if (most_interesting_errno == ENOENT)
 +			most_interesting_errno = errno;
 +	}
 +	errno = most_interesting_errno;
 +	return -1;
 +}
 +
 +/*
 + * Map the loose object at "path" if it is not NULL, or the path found by
 + * searching for a loose object named "sha1".
 + */
 +static void *map_sha1_file_1(struct repository *r, const char *path,
 +			     const unsigned char *sha1, unsigned long *size)
 +{
 +	void *map;
 +	int fd;
 +
 +	if (path)
 +		fd = git_open(path);
 +	else
 +		fd = open_sha1_file(r, sha1, &path);
 +	map = NULL;
 +	if (fd >= 0) {
 +		struct stat st;
 +
 +		if (!fstat(fd, &st)) {
 +			*size = xsize_t(st.st_size);
 +			if (!*size) {
 +				/* mmap() is forbidden on empty files */
 +				error("object file %s is empty", path);
 +				return NULL;
 +			}
 +			map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
 +		}
 +		close(fd);
 +	}
 +	return map;
 +}
 +
 +void *map_sha1_file(struct repository *r,
 +		    const unsigned char *sha1, unsigned long *size)
 +{
 +	return map_sha1_file_1(r, NULL, sha1, size);
 +}
 +
 +static int unpack_sha1_short_header(git_zstream *stream,
 +				    unsigned char *map, unsigned long mapsize,
 +				    void *buffer, unsigned long bufsiz)
 +{
 +	/* Get the data stream */
 +	memset(stream, 0, sizeof(*stream));
 +	stream->next_in = map;
 +	stream->avail_in = mapsize;
 +	stream->next_out = buffer;
 +	stream->avail_out = bufsiz;
 +
 +	git_inflate_init(stream);
 +	return git_inflate(stream, 0);
 +}
 +
 +int unpack_sha1_header(git_zstream *stream,
 +		       unsigned char *map, unsigned long mapsize,
 +		       void *buffer, unsigned long bufsiz)
 +{
 +	int status = unpack_sha1_short_header(stream, map, mapsize,
 +					      buffer, bufsiz);
 +
 +	if (status < Z_OK)
 +		return status;
 +
 +	/* Make sure we have the terminating NUL */
 +	if (!memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
 +		return -1;
 +	return 0;
 +}
 +
 +static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,
 +					unsigned long mapsize, void *buffer,
 +					unsigned long bufsiz, struct strbuf *header)
 +{
 +	int status;
 +
 +	status = unpack_sha1_short_header(stream, map, mapsize, buffer, bufsiz);
 +	if (status < Z_OK)
 +		return -1;
 +
 +	/*
 +	 * Check if entire header is unpacked in the first iteration.
 +	 */
 +	if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
 +		return 0;
 +
 +	/*
 +	 * buffer[0..bufsiz] was not large enough.  Copy the partial
 +	 * result out to header, and then append the result of further
 +	 * reading the stream.
 +	 */
 +	strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
 +	stream->next_out = buffer;
 +	stream->avail_out = bufsiz;
 +
 +	do {
 +		status = git_inflate(stream, 0);
 +		strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
 +		if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
 +			return 0;
 +		stream->next_out = buffer;
 +		stream->avail_out = bufsiz;
 +	} while (status != Z_STREAM_END);
 +	return -1;
 +}
 +
 +static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)
 +{
 +	int bytes = strlen(buffer) + 1;
 +	unsigned char *buf = xmallocz(size);
 +	unsigned long n;
 +	int status = Z_OK;
 +
 +	n = stream->total_out - bytes;
 +	if (n > size)
 +		n = size;
 +	memcpy(buf, (char *) buffer + bytes, n);
 +	bytes = n;
 +	if (bytes <= size) {
 +		/*
 +		 * The above condition must be (bytes <= size), not
 +		 * (bytes < size).  In other words, even though we
 +		 * expect no more output and set avail_out to zero,
 +		 * the input zlib stream may have bytes that express
 +		 * "this concludes the stream", and we *do* want to
 +		 * eat that input.
 +		 *
 +		 * Otherwise we would not be able to test that we
 +		 * consumed all the input to reach the expected size;
 +		 * we also want to check that zlib tells us that all
 +		 * went well with status == Z_STREAM_END at the end.
 +		 */
 +		stream->next_out = buf + bytes;
 +		stream->avail_out = size - bytes;
 +		while (status == Z_OK)
 +			status = git_inflate(stream, Z_FINISH);
 +	}
 +	if (status == Z_STREAM_END && !stream->avail_in) {
 +		git_inflate_end(stream);
 +		return buf;
 +	}
 +
 +	if (status < 0)
 +		error("corrupt loose object '%s'", sha1_to_hex(sha1));
 +	else if (stream->avail_in)
 +		error("garbage at end of loose object '%s'",
 +		      sha1_to_hex(sha1));
 +	free(buf);
 +	return NULL;
 +}
 +
 +/*
 + * We used to just use "sscanf()", but that's actually way
 + * too permissive for what we want to check. So do an anal
 + * object header parse by hand.
 + */
 +static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,
 +			       unsigned int flags)
 +{
 +	const char *type_buf = hdr;
 +	unsigned long size;
 +	int type, type_len = 0;
 +
 +	/*
 +	 * The type can be of any size but is followed by
 +	 * a space.
 +	 */
 +	for (;;) {
 +		char c = *hdr++;
 +		if (!c)
 +			return -1;
 +		if (c == ' ')
 +			break;
 +		type_len++;
 +	}
 +
 +	type = type_from_string_gently(type_buf, type_len, 1);
 +	if (oi->type_name)
 +		strbuf_add(oi->type_name, type_buf, type_len);
 +	/*
 +	 * Set type to 0 if its an unknown object and
 +	 * we're obtaining the type using '--allow-unknown-type'
 +	 * option.
 +	 */
 +	if ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))
 +		type = 0;
 +	else if (type < 0)
 +		die("invalid object type");
 +	if (oi->typep)
 +		*oi->typep = type;
 +
 +	/*
 +	 * The length must follow immediately, and be in canonical
 +	 * decimal format (ie "010" is not valid).
 +	 */
 +	size = *hdr++ - '0';
 +	if (size > 9)
 +		return -1;
 +	if (size) {
 +		for (;;) {
 +			unsigned long c = *hdr - '0';
 +			if (c > 9)
 +				break;
 +			hdr++;
 +			size = size * 10 + c;
 +		}
 +	}
 +
 +	if (oi->sizep)
 +		*oi->sizep = size;
 +
 +	/*
 +	 * The length must be followed by a zero byte
 +	 */
 +	return *hdr ? -1 : type;
 +}
 +
 +int parse_sha1_header(const char *hdr, unsigned long *sizep)
 +{
 +	struct object_info oi = OBJECT_INFO_INIT;
 +
 +	oi.sizep = sizep;
 +	return parse_sha1_header_extended(hdr, &oi, 0);
 +}
 +
 +static int sha1_loose_object_info(struct repository *r,
 +				  const unsigned char *sha1,
 +				  struct object_info *oi, int flags)
 +{
 +	int status = 0;
 +	unsigned long mapsize;
 +	void *map;
 +	git_zstream stream;
 +	char hdr[MAX_HEADER_LEN];
 +	struct strbuf hdrbuf = STRBUF_INIT;
 +	unsigned long size_scratch;
 +
 +	if (oi->delta_base_sha1)
 +		hashclr(oi->delta_base_sha1);
 +
 +	/*
 +	 * If we don't care about type or size, then we don't
 +	 * need to look inside the object at all. Note that we
 +	 * do not optimize out the stat call, even if the
 +	 * caller doesn't care about the disk-size, since our
 +	 * return value implicitly indicates whether the
 +	 * object even exists.
 +	 */
 +	if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
 +		const char *path;
 +		struct stat st;
 +		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 +			return -1;
 +		if (oi->disk_sizep)
 +			*oi->disk_sizep = st.st_size;
 +		return 0;
 +	}
 +
 +	map = map_sha1_file(r, sha1, &mapsize);
 +	if (!map)
 +		return -1;
 +
 +	if (!oi->sizep)
 +		oi->sizep = &size_scratch;
 +
 +	if (oi->disk_sizep)
 +		*oi->disk_sizep = mapsize;
 +	if ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {
 +		if (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)
 +			status = error("unable to unpack %s header with --allow-unknown-type",
 +				       sha1_to_hex(sha1));
 +	} else if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)
 +		status = error("unable to unpack %s header",
 +			       sha1_to_hex(sha1));
 +	if (status < 0)
 +		; /* Do nothing */
 +	else if (hdrbuf.len) {
 +		if ((status = parse_sha1_header_extended(hdrbuf.buf, oi, flags)) < 0)
 +			status = error("unable to parse %s header with --allow-unknown-type",
 +				       sha1_to_hex(sha1));
 +	} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)
 +		status = error("unable to parse %s header", sha1_to_hex(sha1));
 +
 +	if (status >= 0 && oi->contentp) {
 +		*oi->contentp = unpack_sha1_rest(&stream, hdr,
 +						 *oi->sizep, sha1);
 +		if (!*oi->contentp) {
 +			git_inflate_end(&stream);
 +			status = -1;
 +		}
 +	} else
 +		git_inflate_end(&stream);
 +
 +	munmap(map, mapsize);
 +	if (status && oi->typep)
 +		*oi->typep = status;
 +	if (oi->sizep == &size_scratch)
 +		oi->sizep = NULL;
 +	strbuf_release(&hdrbuf);
 +	oi->whence = OI_LOOSE;
 +	return (status < 0) ? status : 0;
 +}
 +
 +int fetch_if_missing = 1;
 +
 +int oid_object_info_extended(const struct object_id *oid, struct object_info *oi, unsigned flags)
 +{
 +	static struct object_info blank_oi = OBJECT_INFO_INIT;
 +	struct pack_entry e;
 +	int rtype;
 +	const struct object_id *real = oid;
 +	int already_retried = 0;
 +
 +	if (flags & OBJECT_INFO_LOOKUP_REPLACE)
 +		real = lookup_replace_object(the_repository, oid);
 +
 +	if (is_null_oid(real))
 +		return -1;
 +
 +	if (!oi)
 +		oi = &blank_oi;
 +
 +	if (!(flags & OBJECT_INFO_SKIP_CACHED)) {
 +		struct cached_object *co = find_cached_object(real->hash);
 +		if (co) {
 +			if (oi->typep)
 +				*(oi->typep) = co->type;
 +			if (oi->sizep)
 +				*(oi->sizep) = co->size;
 +			if (oi->disk_sizep)
 +				*(oi->disk_sizep) = 0;
 +			if (oi->delta_base_sha1)
 +				hashclr(oi->delta_base_sha1);
 +			if (oi->type_name)
 +				strbuf_addstr(oi->type_name, type_name(co->type));
 +			if (oi->contentp)
 +				*oi->contentp = xmemdupz(co->buf, co->size);
 +			oi->whence = OI_CACHED;
 +			return 0;
 +		}
 +	}
 +
 +	while (1) {
 +		if (find_pack_entry(the_repository, real->hash, &e))
 +			break;
 +
 +		if (flags & OBJECT_INFO_IGNORE_LOOSE)
 +			return -1;
 +
 +		/* Most likely it's a loose object. */
 +		if (!sha1_loose_object_info(the_repository, real->hash, oi, flags))
 +			return 0;
 +
 +		/* Not a loose object; someone else may have just packed it. */
 +		if (!(flags & OBJECT_INFO_QUICK)) {
 +			reprepare_packed_git(the_repository);
 +			if (find_pack_entry(the_repository, real->hash, &e))
 +				break;
 +		}
 +
 +		/* Check if it is a missing object */
 +		if (fetch_if_missing && repository_format_partial_clone &&
 +		    !already_retried) {
 +			/*
 +			 * TODO Investigate haveing fetch_object() return
 +			 * TODO error/success and stopping the music here.
 +			 */
 +			fetch_object(repository_format_partial_clone, real->hash);
 +			already_retried = 1;
 +			continue;
 +		}
 +
 +		return -1;
 +	}
 +
 +	if (oi == &blank_oi)
 +		/*
 +		 * We know that the caller doesn't actually need the
 +		 * information below, so return early.
 +		 */
 +		return 0;
 +	rtype = packed_object_info(e.p, e.offset, oi);
 +	if (rtype < 0) {
 +		mark_bad_packed_object(e.p, real->hash);
 +		return oid_object_info_extended(real, oi, 0);
 +	} else if (oi->whence == OI_PACKED) {
 +		oi->u.packed.offset = e.offset;
 +		oi->u.packed.pack = e.p;
 +		oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
 +					 rtype == OBJ_OFS_DELTA);
 +	}
 +
 +	return 0;
 +}
 +
 +/* returns enum object_type or negative */
 +int oid_object_info(const struct object_id *oid, unsigned long *sizep)
 +{
 +	enum object_type type;
 +	struct object_info oi = OBJECT_INFO_INIT;
 +
 +	oi.typep = &type;
 +	oi.sizep = sizep;
 +	if (oid_object_info_extended(oid, &oi,
 +				     OBJECT_INFO_LOOKUP_REPLACE) < 0)
 +		return -1;
 +	return type;
 +}
 +
 +static void *read_object(const unsigned char *sha1, enum object_type *type,
 +			 unsigned long *size)
 +{
 +	struct object_id oid;
 +	struct object_info oi = OBJECT_INFO_INIT;
 +	void *content;
 +	oi.typep = type;
 +	oi.sizep = size;
 +	oi.contentp = &content;
 +
 +	hashcpy(oid.hash, sha1);
 +
 +	if (oid_object_info_extended(&oid, &oi, 0) < 0)
 +		return NULL;
 +	return content;
 +}
 +
 +int pretend_object_file(void *buf, unsigned long len, enum object_type type,
 +			struct object_id *oid)
 +{
 +	struct cached_object *co;
 +
 +	hash_object_file(buf, len, type_name(type), oid);
 +	if (has_sha1_file(oid->hash) || find_cached_object(oid->hash))
 +		return 0;
 +	ALLOC_GROW(cached_objects, cached_object_nr + 1, cached_object_alloc);
 +	co = &cached_objects[cached_object_nr++];
 +	co->size = len;
 +	co->type = type;
 +	co->buf = xmalloc(len);
 +	memcpy(co->buf, buf, len);
 +	hashcpy(co->sha1, oid->hash);
 +	return 0;
 +}
 +
 +/*
 + * This function dies on corrupt objects; the callers who want to
 + * deal with them should arrange to call read_object() and give error
 + * messages themselves.
 + */
 +void *read_object_file_extended(const struct object_id *oid,
 +				enum object_type *type,
 +				unsigned long *size,
 +				int lookup_replace)
 +{
 +	void *data;
 +	const struct packed_git *p;
 +	const char *path;
 +	struct stat st;
 +	const struct object_id *repl = lookup_replace ?
 +		lookup_replace_object(the_repository, oid) : oid;
 +
 +	errno = 0;
 +	data = read_object(repl->hash, type, size);
 +	if (data)
 +		return data;
 +
 +	if (errno && errno != ENOENT)
 +		die_errno("failed to read object %s", oid_to_hex(oid));
 +
 +	/* die if we replaced an object with one that does not exist */
 +	if (repl != oid)
 +		die("replacement %s not found for %s",
 +		    oid_to_hex(repl), oid_to_hex(oid));
 +
 +	if (!stat_sha1_file(the_repository, repl->hash, &st, &path))
 +		die("loose object %s (stored in %s) is corrupt",
 +		    oid_to_hex(repl), path);
 +
 +	if ((p = has_packed_and_bad(repl->hash)) != NULL)
 +		die("packed object %s (stored in %s) is corrupt",
 +		    oid_to_hex(repl), p->pack_name);
 +
 +	return NULL;
 +}
 +
 +void *read_object_with_reference(const struct object_id *oid,
 +				 const char *required_type_name,
 +				 unsigned long *size,
 +				 struct object_id *actual_oid_return)
 +{
 +	enum object_type type, required_type;
 +	void *buffer;
 +	unsigned long isize;
 +	struct object_id actual_oid;
 +
 +	required_type = type_from_string(required_type_name);
 +	oidcpy(&actual_oid, oid);
 +	while (1) {
 +		int ref_length = -1;
 +		const char *ref_type = NULL;
 +
 +		buffer = read_object_file(&actual_oid, &type, &isize);
 +		if (!buffer)
 +			return NULL;
 +		if (type == required_type) {
 +			*size = isize;
 +			if (actual_oid_return)
 +				oidcpy(actual_oid_return, &actual_oid);
 +			return buffer;
 +		}
 +		/* Handle references */
 +		else if (type == OBJ_COMMIT)
 +			ref_type = "tree ";
 +		else if (type == OBJ_TAG)
 +			ref_type = "object ";
 +		else {
 +			free(buffer);
 +			return NULL;
 +		}
 +		ref_length = strlen(ref_type);
 +
 +		if (ref_length + GIT_SHA1_HEXSZ > isize ||
 +		    memcmp(buffer, ref_type, ref_length) ||
 +		    get_oid_hex((char *) buffer + ref_length, &actual_oid)) {
 +			free(buffer);
 +			return NULL;
 +		}
 +		free(buffer);
 +		/* Now we have the ID of the referred-to object in
 +		 * actual_oid.  Check again. */
 +	}
 +}
 +
 +static void write_object_file_prepare(const void *buf, unsigned long len,
 +				      const char *type, struct object_id *oid,
 +				      char *hdr, int *hdrlen)
 +{
 +	git_hash_ctx c;
 +
 +	/* Generate the header */
 +	*hdrlen = xsnprintf(hdr, *hdrlen, "%s %lu", type, len)+1;
 +
 +	/* Sha1.. */
 +	the_hash_algo->init_fn(&c);
 +	the_hash_algo->update_fn(&c, hdr, *hdrlen);
 +	the_hash_algo->update_fn(&c, buf, len);
 +	the_hash_algo->final_fn(oid->hash, &c);
 +}
 +
 +/*
 + * Move the just written object into its final resting place.
 + */
 +int finalize_object_file(const char *tmpfile, const char *filename)
 +{
 +	int ret = 0;
 +
 +	if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
 +		goto try_rename;
 +	else if (link(tmpfile, filename))
 +		ret = errno;
 +
 +	/*
 +	 * Coda hack - coda doesn't like cross-directory links,
 +	 * so we fall back to a rename, which will mean that it
 +	 * won't be able to check collisions, but that's not a
 +	 * big deal.
 +	 *
 +	 * The same holds for FAT formatted media.
 +	 *
 +	 * When this succeeds, we just return.  We have nothing
 +	 * left to unlink.
 +	 */
 +	if (ret && ret != EEXIST) {
 +	try_rename:
 +		if (!rename(tmpfile, filename))
 +			goto out;
 +		ret = errno;
 +	}
 +	unlink_or_warn(tmpfile);
 +	if (ret) {
 +		if (ret != EEXIST) {
 +			return error_errno("unable to write sha1 filename %s", filename);
 +		}
 +		/* FIXME!!! Collision check here ? */
 +	}
 +
 +out:
 +	if (adjust_shared_perm(filename))
 +		return error("unable to set permission to '%s'", filename);
 +	return 0;
 +}
 +
 +static int write_buffer(int fd, const void *buf, size_t len)
 +{
 +	if (write_in_full(fd, buf, len) < 0)
 +		return error_errno("file write error");
 +	return 0;
 +}
 +
 +int hash_object_file(const void *buf, unsigned long len, const char *type,
 +		     struct object_id *oid)
 +{
 +	char hdr[MAX_HEADER_LEN];
 +	int hdrlen = sizeof(hdr);
 +	write_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);
 +	return 0;
 +}
 +
 +/* Finalize a file on disk, and close it. */
 +static void close_sha1_file(int fd)
 +{
 +	if (fsync_object_files)
 +		fsync_or_die(fd, "sha1 file");
 +	if (close(fd) != 0)
 +		die_errno("error when closing sha1 file");
 +}
 +
 +/* Size of directory component, including the ending '/' */
 +static inline int directory_size(const char *filename)
 +{
 +	const char *s = strrchr(filename, '/');
 +	if (!s)
 +		return 0;
 +	return s - filename + 1;
 +}
 +
 +/*
 + * This creates a temporary file in the same directory as the final
 + * 'filename'
 + *
 + * We want to avoid cross-directory filename renames, because those
 + * can have problems on various filesystems (FAT, NFS, Coda).
 + */
 +static int create_tmpfile(struct strbuf *tmp, const char *filename)
 +{
 +	int fd, dirlen = directory_size(filename);
 +
 +	strbuf_reset(tmp);
 +	strbuf_add(tmp, filename, dirlen);
 +	strbuf_addstr(tmp, "tmp_obj_XXXXXX");
 +	fd = git_mkstemp_mode(tmp->buf, 0444);
 +	if (fd < 0 && dirlen && errno == ENOENT) {
 +		/*
 +		 * Make sure the directory exists; note that the contents
 +		 * of the buffer are undefined after mkstemp returns an
 +		 * error, so we have to rewrite the whole buffer from
 +		 * scratch.
 +		 */
 +		strbuf_reset(tmp);
 +		strbuf_add(tmp, filename, dirlen - 1);
 +		if (mkdir(tmp->buf, 0777) && errno != EEXIST)
 +			return -1;
 +		if (adjust_shared_perm(tmp->buf))
 +			return -1;
 +
 +		/* Try again */
 +		strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
 +		fd = git_mkstemp_mode(tmp->buf, 0444);
 +	}
 +	return fd;
 +}
 +
 +static int write_loose_object(const struct object_id *oid, char *hdr,
 +			      int hdrlen, const void *buf, unsigned long len,
 +			      time_t mtime)
 +{
 +	int fd, ret;
 +	unsigned char compressed[4096];
 +	git_zstream stream;
 +	git_hash_ctx c;
 +	struct object_id parano_oid;
 +	static struct strbuf tmp_file = STRBUF_INIT;
 +	static struct strbuf filename = STRBUF_INIT;
 +
 +	strbuf_reset(&filename);
 +	sha1_file_name(the_repository, &filename, oid->hash);
 +
 +	fd = create_tmpfile(&tmp_file, filename.buf);
 +	if (fd < 0) {
 +		if (errno == EACCES)
 +			return error("insufficient permission for adding an object to repository database %s", get_object_directory());
 +		else
 +			return error_errno("unable to create temporary file");
 +	}
 +
 +	/* Set it up */
 +	git_deflate_init(&stream, zlib_compression_level);
 +	stream.next_out = compressed;
 +	stream.avail_out = sizeof(compressed);
 +	the_hash_algo->init_fn(&c);
 +
 +	/* First header.. */
 +	stream.next_in = (unsigned char *)hdr;
 +	stream.avail_in = hdrlen;
 +	while (git_deflate(&stream, 0) == Z_OK)
 +		; /* nothing */
 +	the_hash_algo->update_fn(&c, hdr, hdrlen);
 +
 +	/* Then the data itself.. */
 +	stream.next_in = (void *)buf;
 +	stream.avail_in = len;
 +	do {
 +		unsigned char *in0 = stream.next_in;
 +		ret = git_deflate(&stream, Z_FINISH);
 +		the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
 +		if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
 +			die("unable to write sha1 file");
 +		stream.next_out = compressed;
 +		stream.avail_out = sizeof(compressed);
 +	} while (ret == Z_OK);
 +
 +	if (ret != Z_STREAM_END)
 +		die("unable to deflate new object %s (%d)", oid_to_hex(oid),
 +		    ret);
 +	ret = git_deflate_end_gently(&stream);
 +	if (ret != Z_OK)
 +		die("deflateEnd on object %s failed (%d)", oid_to_hex(oid),
 +		    ret);
 +	the_hash_algo->final_fn(parano_oid.hash, &c);
 +	if (oidcmp(oid, &parano_oid) != 0)
 +		die("confused by unstable object source data for %s",
 +		    oid_to_hex(oid));
 +
 +	close_sha1_file(fd);
 +
 +	if (mtime) {
 +		struct utimbuf utb;
 +		utb.actime = mtime;
 +		utb.modtime = mtime;
 +		if (utime(tmp_file.buf, &utb) < 0)
 +			warning_errno("failed utime() on %s", tmp_file.buf);
 +	}
 +
 +	return finalize_object_file(tmp_file.buf, filename.buf);
 +}
 +
 +static int freshen_loose_object(const unsigned char *sha1)
 +{
 +	return check_and_freshen(sha1, 1);
 +}
 +
 +static int freshen_packed_object(const unsigned char *sha1)
 +{
 +	struct pack_entry e;
 +	if (!find_pack_entry(the_repository, sha1, &e))
 +		return 0;
 +	if (e.p->freshened)
 +		return 1;
 +	if (!freshen_file(e.p->pack_name))
 +		return 0;
 +	e.p->freshened = 1;
 +	return 1;
 +}
 +
 +int write_object_file(const void *buf, unsigned long len, const char *type,
 +		      struct object_id *oid)
 +{
 +	char hdr[MAX_HEADER_LEN];
 +	int hdrlen = sizeof(hdr);
 +
 +	/* Normally if we have it in the pack then we do not bother writing
 +	 * it out into .git/objects/??/?{38} file.
 +	 */
 +	write_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);
 +	if (freshen_packed_object(oid->hash) || freshen_loose_object(oid->hash))
 +		return 0;
 +	return write_loose_object(oid, hdr, hdrlen, buf, len, 0);
 +}
 +
 +int hash_object_file_literally(const void *buf, unsigned long len,
 +			       const char *type, struct object_id *oid,
 +			       unsigned flags)
 +{
 +	char *header;
 +	int hdrlen, status = 0;
 +
 +	/* type string, SP, %lu of the length plus NUL must fit this */
 +	hdrlen = strlen(type) + MAX_HEADER_LEN;
 +	header = xmalloc(hdrlen);
 +	write_object_file_prepare(buf, len, type, oid, header, &hdrlen);
 +
 +	if (!(flags & HASH_WRITE_OBJECT))
 +		goto cleanup;
 +	if (freshen_packed_object(oid->hash) || freshen_loose_object(oid->hash))
 +		goto cleanup;
 +	status = write_loose_object(oid, header, hdrlen, buf, len, 0);
 +
 +cleanup:
 +	free(header);
 +	return status;
 +}
 +
 +int force_object_loose(const struct object_id *oid, time_t mtime)
 +{
 +	void *buf;
 +	unsigned long len;
 +	enum object_type type;
 +	char hdr[MAX_HEADER_LEN];
 +	int hdrlen;
 +	int ret;
 +
 +	if (has_loose_object(oid->hash))
 +		return 0;
 +	buf = read_object(oid->hash, &type, &len);
 +	if (!buf)
 +		return error("cannot read sha1_file for %s", oid_to_hex(oid));
 +	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", type_name(type), len) + 1;
 +	ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime);
 +	free(buf);
 +
 +	return ret;
 +}
 +
 +int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
 +{
 +	struct object_id oid;
 +	if (!startup_info->have_repository)
 +		return 0;
 +	hashcpy(oid.hash, sha1);
 +	return oid_object_info_extended(&oid, NULL,
 +					flags | OBJECT_INFO_SKIP_CACHED) >= 0;
 +}
 +
 +int has_object_file(const struct object_id *oid)
 +{
 +	return has_sha1_file(oid->hash);
 +}
 +
 +int has_object_file_with_flags(const struct object_id *oid, int flags)
 +{
 +	return has_sha1_file_with_flags(oid->hash, flags);
 +}
 +
 +static void check_tree(const void *buf, size_t size)
 +{
 +	struct tree_desc desc;
 +	struct name_entry entry;
 +
 +	init_tree_desc(&desc, buf, size);
 +	while (tree_entry(&desc, &entry))
 +		/* do nothing
 +		 * tree_entry() will die() on malformed entries */
 +		;
 +}
 +
 +static void check_commit(const void *buf, size_t size)
 +{
 +	struct commit c;
 +	memset(&c, 0, sizeof(c));
 +	if (parse_commit_buffer(&c, buf, size))
 +		die("corrupt commit");
 +}
 +
 +static void check_tag(const void *buf, size_t size)
 +{
 +	struct tag t;
 +	memset(&t, 0, sizeof(t));
 +	if (parse_tag_buffer(&t, buf, size))
 +		die("corrupt tag");
 +}
 +
 +static int index_mem(struct object_id *oid, void *buf, size_t size,
 +		     enum object_type type,
 +		     const char *path, unsigned flags)
 +{
 +	int ret, re_allocated = 0;
 +	int write_object = flags & HASH_WRITE_OBJECT;
 +
 +	if (!type)
 +		type = OBJ_BLOB;
 +
 +	/*
 +	 * Convert blobs to git internal format
 +	 */
 +	if ((type == OBJ_BLOB) && path) {
 +		struct strbuf nbuf = STRBUF_INIT;
 +		if (convert_to_git(&the_index, path, buf, size, &nbuf,
 +				   get_conv_flags(flags))) {
 +			buf = strbuf_detach(&nbuf, &size);
 +			re_allocated = 1;
 +		}
 +	}
 +	if (flags & HASH_FORMAT_CHECK) {
 +		if (type == OBJ_TREE)
 +			check_tree(buf, size);
 +		if (type == OBJ_COMMIT)
 +			check_commit(buf, size);
 +		if (type == OBJ_TAG)
 +			check_tag(buf, size);
 +	}
 +
 +	if (write_object)
 +		ret = write_object_file(buf, size, type_name(type), oid);
 +	else
 +		ret = hash_object_file(buf, size, type_name(type), oid);
 +	if (re_allocated)
 +		free(buf);
 +	return ret;
 +}
 +
 +static int index_stream_convert_blob(struct object_id *oid, int fd,
 +				     const char *path, unsigned flags)
 +{
 +	int ret;
 +	const int write_object = flags & HASH_WRITE_OBJECT;
 +	struct strbuf sbuf = STRBUF_INIT;
 +
 +	assert(path);
 +	assert(would_convert_to_git_filter_fd(path));
 +
 +	convert_to_git_filter_fd(&the_index, path, fd, &sbuf,
 +				 get_conv_flags(flags));
 +
 +	if (write_object)
 +		ret = write_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),
 +					oid);
 +	else
 +		ret = hash_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),
 +				       oid);
 +	strbuf_release(&sbuf);
 +	return ret;
 +}
 +
 +static int index_pipe(struct object_id *oid, int fd, enum object_type type,
 +		      const char *path, unsigned flags)
 +{
 +	struct strbuf sbuf = STRBUF_INIT;
 +	int ret;
 +
 +	if (strbuf_read(&sbuf, fd, 4096) >= 0)
 +		ret = index_mem(oid, sbuf.buf, sbuf.len, type, path, flags);
 +	else
 +		ret = -1;
 +	strbuf_release(&sbuf);
 +	return ret;
 +}
 +
 +#define SMALL_FILE_SIZE (32*1024)
 +
 +static int index_core(struct object_id *oid, int fd, size_t size,
 +		      enum object_type type, const char *path,
 +		      unsigned flags)
 +{
 +	int ret;
 +
 +	if (!size) {
 +		ret = index_mem(oid, "", size, type, path, flags);
 +	} else if (size <= SMALL_FILE_SIZE) {
 +		char *buf = xmalloc(size);
 +		ssize_t read_result = read_in_full(fd, buf, size);
 +		if (read_result < 0)
 +			ret = error_errno("read error while indexing %s",
 +					  path ? path : "<unknown>");
 +		else if (read_result != size)
 +			ret = error("short read while indexing %s",
 +				    path ? path : "<unknown>");
 +		else
 +			ret = index_mem(oid, buf, size, type, path, flags);
 +		free(buf);
 +	} else {
 +		void *buf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
 +		ret = index_mem(oid, buf, size, type, path, flags);
 +		munmap(buf, size);
 +	}
 +	return ret;
 +}
 +
 +/*
 + * This creates one packfile per large blob unless bulk-checkin
 + * machinery is "plugged".
 + *
 + * This also bypasses the usual "convert-to-git" dance, and that is on
 + * purpose. We could write a streaming version of the converting
 + * functions and insert that before feeding the data to fast-import
 + * (or equivalent in-core API described above). However, that is
 + * somewhat complicated, as we do not know the size of the filter
 + * result, which we need to know beforehand when writing a git object.
 + * Since the primary motivation for trying to stream from the working
 + * tree file and to avoid mmaping it in core is to deal with large
 + * binary blobs, they generally do not want to get any conversion, and
 + * callers should avoid this code path when filters are requested.
 + */
 +static int index_stream(struct object_id *oid, int fd, size_t size,
 +			enum object_type type, const char *path,
 +			unsigned flags)
 +{
 +	return index_bulk_checkin(oid, fd, size, type, path, flags);
 +}
 +
 +int index_fd(struct object_id *oid, int fd, struct stat *st,
 +	     enum object_type type, const char *path, unsigned flags)
 +{
 +	int ret;
 +
 +	/*
 +	 * Call xsize_t() only when needed to avoid potentially unnecessary
 +	 * die() for large files.
 +	 */
 +	if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(path))
 +		ret = index_stream_convert_blob(oid, fd, path, flags);
 +	else if (!S_ISREG(st->st_mode))
 +		ret = index_pipe(oid, fd, type, path, flags);
 +	else if (st->st_size <= big_file_threshold || type != OBJ_BLOB ||
 +		 (path && would_convert_to_git(&the_index, path)))
 +		ret = index_core(oid, fd, xsize_t(st->st_size), type, path,
 +				 flags);
 +	else
 +		ret = index_stream(oid, fd, xsize_t(st->st_size), type, path,
 +				   flags);
 +	close(fd);
 +	return ret;
 +}
 +
 +int index_path(struct object_id *oid, const char *path, struct stat *st, unsigned flags)
 +{
 +	int fd;
 +	struct strbuf sb = STRBUF_INIT;
 +	int rc = 0;
 +
 +	switch (st->st_mode & S_IFMT) {
 +	case S_IFREG:
 +		fd = open(path, O_RDONLY);
 +		if (fd < 0)
 +			return error_errno("open(\"%s\")", path);
 +		if (index_fd(oid, fd, st, OBJ_BLOB, path, flags) < 0)
 +			return error("%s: failed to insert into database",
 +				     path);
 +		break;
 +	case S_IFLNK:
 +		if (strbuf_readlink(&sb, path, st->st_size))
 +			return error_errno("readlink(\"%s\")", path);
 +		if (!(flags & HASH_WRITE_OBJECT))
 +			hash_object_file(sb.buf, sb.len, blob_type, oid);
 +		else if (write_object_file(sb.buf, sb.len, blob_type, oid))
 +			rc = error("%s: failed to insert into database", path);
 +		strbuf_release(&sb);
 +		break;
 +	case S_IFDIR:
 +		return resolve_gitlink_ref(path, "HEAD", oid);
 +	default:
 +		return error("%s: unsupported file type", path);
 +	}
 +	return rc;
 +}
 +
 +int read_pack_header(int fd, struct pack_header *header)
 +{
 +	if (read_in_full(fd, header, sizeof(*header)) != sizeof(*header))
 +		/* "eof before pack header was fully read" */
 +		return PH_ERROR_EOF;
 +
 +	if (header->hdr_signature != htonl(PACK_SIGNATURE))
 +		/* "protocol error (pack signature mismatch detected)" */
 +		return PH_ERROR_PACK_SIGNATURE;
 +	if (!pack_version_ok(header->hdr_version))
 +		/* "protocol error (pack version unsupported)" */
 +		return PH_ERROR_PROTOCOL;
 +	return 0;
 +}
 +
 +void assert_oid_type(const struct object_id *oid, enum object_type expect)
 +{
 +	enum object_type type = oid_object_info(oid, NULL);
 +	if (type < 0)
 +		die("%s is not a valid object", oid_to_hex(oid));
 +	if (type != expect)
 +		die("%s is not a valid '%s' object", oid_to_hex(oid),
 +		    type_name(expect));
 +}
 +
 +int for_each_file_in_obj_subdir(unsigned int subdir_nr,
 +				struct strbuf *path,
 +				each_loose_object_fn obj_cb,
 +				each_loose_cruft_fn cruft_cb,
 +				each_loose_subdir_fn subdir_cb,
 +				void *data)
 +{
 +	size_t origlen, baselen;
 +	DIR *dir;
 +	struct dirent *de;
 +	int r = 0;
 +	struct object_id oid;
 +
 +	if (subdir_nr > 0xff)
 +		BUG("invalid loose object subdirectory: %x", subdir_nr);
 +
 +	origlen = path->len;
 +	strbuf_complete(path, '/');
 +	strbuf_addf(path, "%02x", subdir_nr);
 +
 +	dir = opendir(path->buf);
 +	if (!dir) {
 +		if (errno != ENOENT)
 +			r = error_errno("unable to open %s", path->buf);
 +		strbuf_setlen(path, origlen);
 +		return r;
 +	}
 +
 +	oid.hash[0] = subdir_nr;
 +	strbuf_addch(path, '/');
 +	baselen = path->len;
 +
 +	while ((de = readdir(dir))) {
 +		size_t namelen;
 +		if (is_dot_or_dotdot(de->d_name))
 +			continue;
 +
 +		namelen = strlen(de->d_name);
 +		strbuf_setlen(path, baselen);
 +		strbuf_add(path, de->d_name, namelen);
 +		if (namelen == GIT_SHA1_HEXSZ - 2 &&
 +		    !hex_to_bytes(oid.hash + 1, de->d_name,
 +				  GIT_SHA1_RAWSZ - 1)) {
 +			if (obj_cb) {
 +				r = obj_cb(&oid, path->buf, data);
 +				if (r)
 +					break;
 +			}
 +			continue;
 +		}
 +
 +		if (cruft_cb) {
 +			r = cruft_cb(de->d_name, path->buf, data);
 +			if (r)
 +				break;
 +		}
 +	}
 +	closedir(dir);
 +
 +	strbuf_setlen(path, baselen - 1);
 +	if (!r && subdir_cb)
 +		r = subdir_cb(subdir_nr, path->buf, data);
 +
 +	strbuf_setlen(path, origlen);
 +
 +	return r;
 +}
 +
 +int for_each_loose_file_in_objdir_buf(struct strbuf *path,
 +			    each_loose_object_fn obj_cb,
 +			    each_loose_cruft_fn cruft_cb,
 +			    each_loose_subdir_fn subdir_cb,
 +			    void *data)
 +{
 +	int r = 0;
 +	int i;
 +
 +	for (i = 0; i < 256; i++) {
 +		r = for_each_file_in_obj_subdir(i, path, obj_cb, cruft_cb,
 +						subdir_cb, data);
 +		if (r)
 +			break;
 +	}
 +
 +	return r;
 +}
 +
 +int for_each_loose_file_in_objdir(const char *path,
 +				  each_loose_object_fn obj_cb,
 +				  each_loose_cruft_fn cruft_cb,
 +				  each_loose_subdir_fn subdir_cb,
 +				  void *data)
 +{
 +	struct strbuf buf = STRBUF_INIT;
 +	int r;
 +
 +	strbuf_addstr(&buf, path);
 +	r = for_each_loose_file_in_objdir_buf(&buf, obj_cb, cruft_cb,
 +					      subdir_cb, data);
 +	strbuf_release(&buf);
 +
 +	return r;
 +}
 +
 +struct loose_alt_odb_data {
 +	each_loose_object_fn *cb;
 +	void *data;
 +};
 +
 +static int loose_from_alt_odb(struct alternate_object_database *alt,
 +			      void *vdata)
 +{
 +	struct loose_alt_odb_data *data = vdata;
 +	struct strbuf buf = STRBUF_INIT;
 +	int r;
 +
 +	strbuf_addstr(&buf, alt->path);
 +	r = for_each_loose_file_in_objdir_buf(&buf,
 +					      data->cb, NULL, NULL,
 +					      data->data);
 +	strbuf_release(&buf);
 +	return r;
 +}
 +
 +int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)
 +{
 +	struct loose_alt_odb_data alt;
 +	int r;
 +
 +	r = for_each_loose_file_in_objdir(get_object_directory(),
 +					  cb, NULL, NULL, data);
 +	if (r)
 +		return r;
 +
 +	if (flags & FOR_EACH_OBJECT_LOCAL_ONLY)
 +		return 0;
 +
 +	alt.cb = cb;
 +	alt.data = data;
 +	return foreach_alt_odb(loose_from_alt_odb, &alt);
 +}
 +
 +static int check_stream_sha1(git_zstream *stream,
 +			     const char *hdr,
 +			     unsigned long size,
 +			     const char *path,
 +			     const unsigned char *expected_sha1)
 +{
 +	git_hash_ctx c;
 +	unsigned char real_sha1[GIT_MAX_RAWSZ];
 +	unsigned char buf[4096];
 +	unsigned long total_read;
 +	int status = Z_OK;
 +
 +	the_hash_algo->init_fn(&c);
 +	the_hash_algo->update_fn(&c, hdr, stream->total_out);
 +
 +	/*
 +	 * We already read some bytes into hdr, but the ones up to the NUL
 +	 * do not count against the object's content size.
 +	 */
 +	total_read = stream->total_out - strlen(hdr) - 1;
 +
 +	/*
 +	 * This size comparison must be "<=" to read the final zlib packets;
 +	 * see the comment in unpack_sha1_rest for details.
 +	 */
 +	while (total_read <= size &&
 +	       (status == Z_OK || status == Z_BUF_ERROR)) {
 +		stream->next_out = buf;
 +		stream->avail_out = sizeof(buf);
 +		if (size - total_read < stream->avail_out)
 +			stream->avail_out = size - total_read;
 +		status = git_inflate(stream, Z_FINISH);
 +		the_hash_algo->update_fn(&c, buf, stream->next_out - buf);
 +		total_read += stream->next_out - buf;
 +	}
 +	git_inflate_end(stream);
 +
 +	if (status != Z_STREAM_END) {
 +		error("corrupt loose object '%s'", sha1_to_hex(expected_sha1));
 +		return -1;
 +	}
 +	if (stream->avail_in) {
 +		error("garbage at end of loose object '%s'",
 +		      sha1_to_hex(expected_sha1));
 +		return -1;
 +	}
 +
 +	the_hash_algo->final_fn(real_sha1, &c);
 +	if (hashcmp(expected_sha1, real_sha1)) {
 +		error("sha1 mismatch for %s (expected %s)", path,
 +		      sha1_to_hex(expected_sha1));
 +		return -1;
 +	}
 +
 +	return 0;
 +}
 +
 +int read_loose_object(const char *path,
 +		      const struct object_id *expected_oid,
 +		      enum object_type *type,
 +		      unsigned long *size,
 +		      void **contents)
 +{
 +	int ret = -1;
 +	void *map = NULL;
 +	unsigned long mapsize;
 +	git_zstream stream;
 +	char hdr[MAX_HEADER_LEN];
 +
 +	*contents = NULL;
 +
 +	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 +	if (!map) {
 +		error_errno("unable to mmap %s", path);
 +		goto out;
 +	}
 +
 +	if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {
 +		error("unable to unpack header of %s", path);
 +		goto out;
 +	}
 +
 +	*type = parse_sha1_header(hdr, size);
 +	if (*type < 0) {
 +		error("unable to parse header of %s", path);
 +		git_inflate_end(&stream);
 +		goto out;
 +	}
 +
 +	if (*type == OBJ_BLOB) {
 +		if (check_stream_sha1(&stream, hdr, *size, path, expected_oid->hash) < 0)
 +			goto out;
 +	} else {
 +		*contents = unpack_sha1_rest(&stream, hdr, *size, expected_oid->hash);
 +		if (!*contents) {
 +			error("unable to unpack contents of %s", path);
 +			git_inflate_end(&stream);
 +			goto out;
 +		}
 +		if (check_object_signature(expected_oid, *contents,
 +					 *size, type_name(*type))) {
 +			error("sha1 mismatch for %s (expected %s)", path,
 +			      oid_to_hex(expected_oid));
 +			free(*contents);
 +			goto out;
 +		}
 +	}
 +
 +	ret = 0; /* everything checks out */
 +
 +out:
 +	if (map)
 +		munmap(map, mapsize);
 +	return ret;
 +}
diff --combined strbuf.c
index 43a840c67b,0a24c3dd76..622c462d54
--- a/strbuf.c
+++ b/strbuf.c
@@@ -11,6 -11,15 +11,15 @@@ int starts_with(const char *str, const 
  			return 0;
  }
  
+ int istarts_with(const char *str, const char *prefix)
+ {
+ 	for (; ; str++, prefix++)
+ 		if (!*prefix)
+ 			return 1;
+ 		else if (tolower(*str) != tolower(*prefix))
+ 			return 0;
+ }
+ 
  int skip_to_optional_arg_default(const char *str, const char *prefix,
  				 const char **arg, const char *def)
  {
@@@ -95,7 -104,6 +104,7 @@@ void strbuf_trim(struct strbuf *sb
  	strbuf_rtrim(sb);
  	strbuf_ltrim(sb);
  }
 +
  void strbuf_rtrim(struct strbuf *sb)
  {
  	while (sb->len > 0 && isspace((unsigned char)sb->buf[sb->len - 1]))
@@@ -103,13 -111,6 +112,13 @@@
  	sb->buf[sb->len] = '\0';
  }
  
 +void strbuf_trim_trailing_dir_sep(struct strbuf *sb)
 +{
 +	while (sb->len > 0 && is_dir_sep((unsigned char)sb->buf[sb->len - 1]))
 +		sb->len--;
 +	sb->buf[sb->len] = '\0';
 +}
 +
  void strbuf_ltrim(struct strbuf *sb)
  {
  	char *b = sb->buf;
@@@ -620,18 -621,14 +629,18 @@@ ssize_t strbuf_read_file(struct strbuf 
  {
  	int fd;
  	ssize_t len;
 +	int saved_errno;
  
  	fd = open(path, O_RDONLY);
  	if (fd < 0)
  		return -1;
  	len = strbuf_read(sb, fd, hint);
 +	saved_errno = errno;
  	close(fd);
 -	if (len < 0)
 +	if (len < 0) {
 +		errno = saved_errno;
  		return -1;
 +	}
  
  	return len;
  }
@@@ -695,7 -692,7 +704,7 @@@ static void strbuf_add_urlencode(struc
  		    (!reserved && is_rfc3986_reserved(ch)))
  			strbuf_addch(sb, ch);
  		else
 -			strbuf_addf(sb, "%%%02x", ch);
 +			strbuf_addf(sb, "%%%02x", (unsigned char)ch);
  	}
  }
  
@@@ -793,7 -790,18 +802,18 @@@ char *xstrdup_tolower(const char *strin
  	result = xmallocz(len);
  	for (i = 0; i < len; i++)
  		result[i] = tolower(string[i]);
- 	result[i] = '\0';
+ 	return result;
+ }
+ 
+ char *xstrdup_toupper(const char *string)
+ {
+ 	char *result;
+ 	size_t len, i;
+ 
+ 	len = strlen(string);
+ 	result = xmallocz(len);
+ 	for (i = 0; i < len; i++)
+ 		result[i] = toupper(string[i]);
  	return result;
  }
  
@@@ -881,12 -889,12 +901,12 @@@ void strbuf_addftime(struct strbuf *sb
  	strbuf_setlen(sb, sb->len + len);
  }
  
 -void strbuf_add_unique_abbrev(struct strbuf *sb, const unsigned char *sha1,
 +void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,
  			      int abbrev_len)
  {
  	int r;
  	strbuf_grow(sb, GIT_SHA1_HEXSZ + 1);
 -	r = find_unique_abbrev_r(sb->buf + sb->len, sha1, abbrev_len);
 +	r = find_unique_abbrev_r(sb->buf + sb->len, oid, abbrev_len);
  	strbuf_setlen(sb, sb->len + r);
  }
  
diff --combined strbuf.h
index 4efa80c1de,df7ced53ed..8c25e4bb59
--- a/strbuf.h
+++ b/strbuf.h
@@@ -70,12 -70,6 +70,12 @@@ struct strbuf 
  extern char strbuf_slopbuf[];
  #define STRBUF_INIT  { .alloc = 0, .len = 0, .buf = strbuf_slopbuf }
  
 +/*
 + * Predeclare this here, since cache.h includes this file before it defines the
 + * struct.
 + */
 +struct object_id;
 +
  /**
   * Life Cycle Functions
   * --------------------
@@@ -185,9 -179,6 +185,9 @@@ extern void strbuf_trim(struct strbuf *
  extern void strbuf_rtrim(struct strbuf *);
  extern void strbuf_ltrim(struct strbuf *);
  
 +/* Strip trailing directory separators */
 +extern void strbuf_trim_trailing_dir_sep(struct strbuf *);
 +
  /**
   * Replace the contents of the strbuf with a reencoded form.  Returns -1
   * on error, 0 on success.
@@@ -548,7 -539,7 +548,7 @@@ extern void strbuf_list_free(struct str
   * the strbuf `sb`.
   */
  extern void strbuf_add_unique_abbrev(struct strbuf *sb,
 -				     const unsigned char *sha1,
 +				     const struct object_id *oid,
  				     int abbrev_len);
  
  /**
@@@ -616,6 -607,7 +616,7 @@@ __attribute__((format (printf,2,3))
  extern int fprintf_ln(FILE *fp, const char *fmt, ...);
  
  char *xstrdup_tolower(const char *);
+ char *xstrdup_toupper(const char *);
  
  /**
   * Create a newly allocated string using printf format. You can do this easily
diff --combined utf8.c
index 4419055b48,25d366d6b3..0fcc6487e3
--- a/utf8.c
+++ b/utf8.c
@@@ -81,7 -81,7 +81,7 @@@ static int git_wcwidth(ucs_char_t ch
  	/*
  	 * Sorted list of non-overlapping intervals of non-spacing characters,
  	 */
 -#include "unicode_width.h"
 +#include "unicode-width.h"
  
  	/* test for 8-bit control characters */
  	if (ch == 0)
@@@ -401,18 -401,40 +401,40 @@@ out
  	strbuf_release(&sb_dst);
  }
  
+ /*
+  * Returns true (1) if the src encoding name matches the dst encoding
+  * name directly or one of its alternative names. E.g. UTF-16BE is the
+  * same as UTF16BE.
+  */
+ static int same_utf_encoding(const char *src, const char *dst)
+ {
+ 	if (istarts_with(src, "utf") && istarts_with(dst, "utf")) {
+ 		/* src[3] or dst[3] might be '\0' */
+ 		int i = (src[3] == '-' ? 4 : 3);
+ 		int j = (dst[3] == '-' ? 4 : 3);
+ 		return !strcasecmp(src+i, dst+j);
+ 	}
+ 	return 0;
+ }
+ 
  int is_encoding_utf8(const char *name)
  {
  	if (!name)
  		return 1;
- 	if (!strcasecmp(name, "utf-8") || !strcasecmp(name, "utf8"))
+ 	if (same_utf_encoding("utf-8", name))
  		return 1;
  	return 0;
  }
  
  int same_encoding(const char *src, const char *dst)
  {
- 	if (is_encoding_utf8(src) && is_encoding_utf8(dst))
+ 	static const char utf8[] = "UTF-8";
+ 
+ 	if (!src)
+ 		src = utf8;
+ 	if (!dst)
+ 		dst = utf8;
+ 	if (same_utf_encoding(src, dst))
  		return 1;
  	return !strcasecmp(src, dst);
  }
@@@ -538,6 -560,45 +560,45 @@@ char *reencode_string_len(const char *i
  }
  #endif
  
+ static int has_bom_prefix(const char *data, size_t len,
+ 			  const char *bom, size_t bom_len)
+ {
+ 	return data && bom && (len >= bom_len) && !memcmp(data, bom, bom_len);
+ }
+ 
+ static const char utf16_be_bom[] = {0xFE, 0xFF};
+ static const char utf16_le_bom[] = {0xFF, 0xFE};
+ static const char utf32_be_bom[] = {0x00, 0x00, 0xFE, 0xFF};
+ static const char utf32_le_bom[] = {0xFF, 0xFE, 0x00, 0x00};
+ 
+ int has_prohibited_utf_bom(const char *enc, const char *data, size_t len)
+ {
+ 	return (
+ 	  (same_utf_encoding("UTF-16BE", enc) ||
+ 	   same_utf_encoding("UTF-16LE", enc)) &&
+ 	  (has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||
+ 	   has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))
+ 	) || (
+ 	  (same_utf_encoding("UTF-32BE",  enc) ||
+ 	   same_utf_encoding("UTF-32LE", enc)) &&
+ 	  (has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||
+ 	   has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))
+ 	);
+ }
+ 
+ int is_missing_required_utf_bom(const char *enc, const char *data, size_t len)
+ {
+ 	return (
+ 	   (same_utf_encoding(enc, "UTF-16")) &&
+ 	   !(has_bom_prefix(data, len, utf16_be_bom, sizeof(utf16_be_bom)) ||
+ 	     has_bom_prefix(data, len, utf16_le_bom, sizeof(utf16_le_bom)))
+ 	) || (
+ 	   (same_utf_encoding(enc, "UTF-32")) &&
+ 	   !(has_bom_prefix(data, len, utf32_be_bom, sizeof(utf32_be_bom)) ||
+ 	     has_bom_prefix(data, len, utf32_le_bom, sizeof(utf32_le_bom)))
+ 	);
+ }
+ 
  /*
   * Returns first character length in bytes for multi-byte `text` according to
   * `encoding`.