From: Junio C Hamano Date: Tue, 17 Jan 2017 22:49:29 +0000 (-0800) Subject: Merge branch 'jk/http-walker-limit-redirect' into maint X-Git-Tag: v2.11.1~55 X-Git-Url: https://git.lorimer.id.au/gitweb.git/diff_plain/5ce6f51ff78bcb847c6c4835d80f6b58d730a8de?hp=-c Merge branch 'jk/http-walker-limit-redirect' into maint Update the error messages from the dumb-http client when it fails to obtain loose objects; we used to give sensible error message only upon 404 but we now forbid unexpected redirects that needs to be reported with something sensible. * jk/http-walker-limit-redirect: http-walker: complain about non-404 loose object errors http: treat http-alternates like redirects http: make redirects more obvious remote-curl: rename shadowed options variable http: always update the base URL for redirects http: simplify update_url_from_redirect --- 5ce6f51ff78bcb847c6c4835d80f6b58d730a8de diff --combined Documentation/config.txt index a0ab66aae7,8153336435..d51182a060 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@@ -150,34 -150,27 +150,34 @@@ integer: 1024", "by 1024x1024", etc. color:: - The value for a variables that takes a color is a list of - colors (at most two) and attributes (at most one), separated - by spaces. The colors accepted are `normal`, `black`, - `red`, `green`, `yellow`, `blue`, `magenta`, `cyan` and - `white`; the attributes are `bold`, `dim`, `ul`, `blink` and - `reverse`. The first color given is the foreground; the - second is the background. The position of the attribute, if - any, doesn't matter. Attributes may be turned off specifically - by prefixing them with `no` (e.g., `noreverse`, `noul`, etc). -+ -Colors (foreground and background) may also be given as numbers between -0 and 255; these use ANSI 256-color mode (but note that not all -terminals may support this). If your terminal supports it, you may also -specify 24-bit RGB values as hex, like `#ff0ab3`. -+ -The attributes are meant to be reset at the beginning of each item -in the colored output, so setting color.decorate.branch to `black` -will paint that branch name in a plain `black`, even if the previous -thing on the same output line (e.g. opening parenthesis before the -list of branch names in `log --decorate` output) is set to be -painted with `bold` or some other attribute. + The value for a variable that takes a color is a list of + colors (at most two, one for foreground and one for background) + and attributes (as many as you want), separated by spaces. ++ +The basic colors accepted are `normal`, `black`, `red`, `green`, `yellow`, +`blue`, `magenta`, `cyan` and `white`. The first color given is the +foreground; the second is the background. ++ +Colors may also be given as numbers between 0 and 255; these use ANSI +256-color mode (but note that not all terminals may support this). If +your terminal supports it, you may also specify 24-bit RGB values as +hex, like `#ff0ab3`. ++ +The accepted attributes are `bold`, `dim`, `ul`, `blink`, `reverse`, +`italic`, and `strike` (for crossed-out or "strikethrough" letters). +The position of any attributes with respect to the colors +(before, after, or in between), doesn't matter. Specific attributes may +be turned off by prefixing them with `no` or `no-` (e.g., `noreverse`, +`no-ul`, etc). ++ +For git's pre-defined color slots, the attributes are meant to be reset +at the beginning of each item in the colored output. So setting +`color.decorate.branch` to `black` will paint that branch name in a +plain `black`, even if the previous thing on the same output line (e.g. +opening parenthesis before the list of branch names in `log --decorate` +output) is set to be painted with `bold` or some other attribute. +However, custom log formats may do more complicated and layered +coloring, and the negated forms may be useful there. pathname:: A variable that takes a pathname value can be given a @@@ -448,13 -441,6 +448,13 @@@ specify that no proxy be used for a giv This is useful for excluding servers inside a firewall from proxy use, while defaulting to a common proxy for external domains. +core.sshCommand:: + If this variable is set, `git fetch` and `git push` will + use the specified command instead of `ssh` when they need to + connect to a remote system. The command is in the same form as + the `GIT_SSH_COMMAND` environment variable and is overridden + when the environment variable is set. + core.ignoreStat:: If true, Git will avoid using lstat() calls to detect if files have changed by setting the "assume-unchanged" bit for those tracked files @@@ -953,8 -939,7 +953,8 @@@ color.branch: A boolean to enable/disable color in the output of linkgit:git-branch[1]. May be set to `always`, `false` (or `never`) or `auto` (or `true`), in which case colors are used - only when the output is to a terminal. Defaults to false. + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default). color.branch.:: Use customized color for branch coloration. `` is one of @@@ -969,8 -954,7 +969,8 @@@ color.diff: linkgit:git-log[1], and linkgit:git-show[1] will use color for all patches. If it is set to `true` or `auto`, those commands will only use color when output is to the terminal. - Defaults to false. + If unset, then the value of `color.ui` is used (`auto` by + default). + This does not affect linkgit:git-format-patch[1] or the 'git-diff-{asterisk}' plumbing commands. Can be overridden on the @@@ -993,8 -977,7 +993,8 @@@ color.decorate.: color.grep:: When set to `always`, always highlight matches. When `false` (or `never`), never. When set to `true` or `auto`, use color only - when the output is written to the terminal. Defaults to `false`. + when the output is written to the terminal. If unset, then the + value of `color.ui` is used (`auto` by default). color.grep.:: Use customized color for grep colorization. `` specifies which @@@ -1027,8 -1010,7 +1027,8 @@@ color.interactive: and displays (such as those used by "git-add --interactive" and "git-clean --interactive"). When false (or `never`), never. When set to `true` or `auto`, use colors only when the output is - to the terminal. Defaults to false. + to the terminal. If unset, then the value of `color.ui` is + used (`auto` by default). color.interactive.:: Use customized color for 'git add --interactive' and 'git clean @@@ -1044,15 -1026,13 +1044,15 @@@ color.showBranch: A boolean to enable/disable color in the output of linkgit:git-show-branch[1]. May be set to `always`, `false` (or `never`) or `auto` (or `true`), in which case colors are used - only when the output is to a terminal. Defaults to false. + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default). color.status:: A boolean to enable/disable color in the output of linkgit:git-status[1]. May be set to `always`, `false` (or `never`) or `auto` (or `true`), in which case colors are used - only when the output is to a terminal. Defaults to false. + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default). color.status.:: Use customized color for status colorization. `` is @@@ -1207,15 -1187,6 +1207,15 @@@ difftool..cmd: difftool.prompt:: Prompt before each invocation of the diff tool. +fastimport.unpackLimit:: + If the number of objects imported by linkgit:git-fast-import[1] + is below this limit, then the objects will be unpacked into + loose object files. However if the number of imported objects + equals or exceeds this limit then the pack will be stored as a + pack. Storing the pack from a fast-import can make the import + operation complete faster, especially on slow filesystems. If + not set, the value of `transfer.unpackLimit` is used instead. + fetch.recurseSubmodules:: This option can be either set to a boolean value or to 'on-demand'. Setting it to a boolean changes the behavior of fetch and pull to @@@ -1247,11 -1218,6 +1247,11 @@@ fetch.prune: If true, fetch will automatically behave as if the `--prune` option was given on the command line. See also `remote..prune`. +fetch.output:: + Control how ref update status is printed. Valid values are + `full` and `compact`. Default value is `full`. See section + OUTPUT in linkgit:git-fetch[1] for detail. + format.attach:: Enable multipart/mixed attachments as the default for 'format-patch'. The value can also be a double quoted string @@@ -1259,16 -1225,6 +1259,16 @@@ value as the boundary. See the --attach option in linkgit:git-format-patch[1]. +format.from:: + Provides the default value for the `--from` option to format-patch. + Accepts a boolean value, or a name and email address. If false, + format-patch defaults to `--no-from`, using commit authors directly in + the "From:" field of patch mails. If true, format-patch defaults to + `--from`, using your committer identity in the "From:" field of patch + mails and including a "From:" field in the body of the patch mail if + different. If set to a non-boolean value, format-patch uses that + value instead of your committer identity. Defaults to false. + format.numbered:: A boolean which can enable or disable sequence numbers in patch subjects. It defaults to "auto" which enables it only if there @@@ -1372,7 -1328,7 +1372,7 @@@ fsck.skipList: gc.aggressiveDepth:: The depth parameter used in the delta compression algorithm used by 'git gc --aggressive'. This defaults - to 250. + to 50. gc.aggressiveWindow:: The window size parameter used in the delta compression @@@ -1736,20 -1692,6 +1736,20 @@@ http.emptyAuth: a username in the URL, as libcurl normally requires a username for authentication. +http.delegation:: + Control GSSAPI credential delegation. The delegation is disabled + by default in libcurl since version 7.21.7. Set parameter to tell + the server what it is allowed to delegate when it comes to user + credentials. Used with GSS/kerberos. Possible values are: ++ +-- +* `none` - Don't allow any delegation. +* `policy` - Delegates if and only if the OK-AS-DELEGATE flag is set in the + Kerberos service ticket, which is a matter of realm policy. +* `always` - Unconditionally allow the server to delegate. +-- + + http.extraHeader:: Pass an additional HTTP header when communicating with a server. If more than one such entry exists, all of them are added as extra @@@ -1891,6 -1833,16 +1891,16 @@@ http.userAgent: of common USER_AGENT strings (but not including those like git/1.7.1). Can be overridden by the `GIT_HTTP_USER_AGENT` environment variable. + http.followRedirects:: + Whether git should follow HTTP redirects. If set to `true`, git + will transparently follow any redirect issued by a server it + encounters. If set to `false`, git will treat all redirects as + errors. If set to `initial`, git will follow redirects only for + the initial request to a remote, but not for subsequent + follow-up HTTP requests. Since git uses the redirected URL as + the base for the follow-up requests, this is generally + sufficient. The default is `initial`. + http..*:: Any of the http.* options above can be applied selectively to some URLs. For a config key to match a URL, each element of the config key is @@@ -2450,20 -2402,15 +2460,20 @@@ rebase.missingCommitsCheck: command in the todo-list. Defaults to "ignore". -rebase.instructionFormat +rebase.instructionFormat:: A format string, as specified in linkgit:git-log[1], to be used for the instruction list during an interactive rebase. The format will automatically have the long commit hash prepended to the format. receive.advertiseAtomic:: By default, git-receive-pack will advertise the atomic push - capability to its clients. If you don't want to this capability - to be advertised, set this variable to false. + capability to its clients. If you don't want to advertise this + capability, set this variable to false. + +receive.advertisePushOptions:: + By default, git-receive-pack will advertise the push options + capability to its clients. If you don't want to advertise this + capability, set this variable to false. receive.autogc:: By default, git-receive-pack will run "git-gc --auto" after @@@ -2518,15 -2465,6 +2528,15 @@@ receive.fsck.skipList: can be safely ignored such as invalid committer email addresses. Note: corrupt objects cannot be skipped with this setting. +receive.keepAlive:: + After receiving the pack from the client, `receive-pack` may + produce no output (if `--quiet` was specified) while processing + the pack, causing some networks to drop the TCP connection. + With this option set, if `receive-pack` does not transmit + any data in this phase for `receive.keepAlive` seconds, it will + send a short keepalive packet. The default is 5 seconds; set + to 0 to disable keepalives entirely. + receive.unpackLimit:: If the number of objects received in a push is below this limit then the objects will be unpacked into loose object @@@ -2537,12 -2475,6 +2547,12 @@@ especially on slow filesystems. If not set, the value of `transfer.unpackLimit` is used instead. +receive.maxInputSize:: + If the size of the incoming pack stream is larger than this + limit, then git-receive-pack will error out, instead of + accepting the pack file. If not set or set to 0, then the size + is unlimited. + receive.denyDeletes:: If set to true, git-receive-pack will deny a ref update that deletes the ref. Use this to prevent such a ref deletion via a push. @@@ -2825,13 -2757,12 +2835,13 @@@ stash.showStat: option will show diffstat of the stash. Defaults to true. See description of 'show' command in linkgit:git-stash[1]. -submodule..path:: submodule..url:: - The path within this project and URL for a submodule. These - variables are initially populated by 'git submodule init'. See - linkgit:git-submodule[1] and linkgit:gitmodules[5] for - details. + The URL for a submodule. This variable is copied from the .gitmodules + file to the git config via 'git submodule init'. The user can change + the configured URL before obtaining the submodule via 'git submodule + update'. After obtaining the submodule, the presence of this variable + is used as a sign whether the submodule is of interest to git commands. + See linkgit:git-submodule[1] and linkgit:gitmodules[5] for details. submodule..update:: The default update procedure for a submodule. This variable @@@ -2874,18 -2805,6 +2884,18 @@@ submodule.fetchJobs: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.alternateLocation:: + Specifies how the submodules obtain alternates when submodules are + cloned. Possible values are `no`, `superproject`. + By default `no` is assumed, which doesn't add references. When the + value is set to `superproject` the submodule to be cloned computes + its alternates location relative to the superprojects alternate. + +submodule.alternateErrorStrategy + Specifies how to treat errors with the alternates for a submodule + as computed via `submodule.alternateLocation`. Possible values are + `ignore`, `info`, `die`. Default is `die`. + tag.forceSignAnnotated:: A boolean to specify whether annotated tags created should be GPG signed. If `--annotate` is specified on the command line, it takes @@@ -2972,21 -2891,6 +2982,21 @@@ uploadpack.keepAlive: `uploadpack.keepAlive` seconds. Setting this option to 0 disables keepalive packets entirely. The default is 5 seconds. +uploadpack.packObjectsHook:: + If this option is set, when `upload-pack` would run + `git pack-objects` to create a packfile for a client, it will + run this shell command instead. The `pack-objects` command and + arguments it _would_ have run (including the `git pack-objects` + at the beginning) are appended to the shell command. The stdin + and stdout of the hook are treated as if `pack-objects` itself + was run. I.e., `upload-pack` will feed input intended for + `pack-objects` to the hook, and expects a completed packfile on + stdout. ++ +Note that this configuration variable is ignored if it is seen in the +repository-level config (this is a safety measure against fetching from +untrusted repositories). + url..insteadOf:: Any URL that starts with this value will be rewritten to start, instead, with . In cases where some site serves a diff --combined http.c index 4c4a812fcc,5cd3ffd67f..051fe6e5ab --- a/http.c +++ b/http.c @@@ -11,7 -11,6 +11,7 @@@ #include "gettext.h" #include "transport.h" +static struct trace_key trace_curl = TRACE_KEY_INIT(CURL); #if LIBCURL_VERSION_NUM >= 0x070a08 long int git_curl_ipresolve = CURL_IPRESOLVE_WHATEVER; #else @@@ -90,18 -89,6 +90,18 @@@ static struct * here, too */ }; +#if LIBCURL_VERSION_NUM >= 0x071600 +static const char *curl_deleg; +static struct { + const char *name; + long curl_deleg_param; +} curl_deleg_levels[] = { + { "none", CURLGSSAPI_DELEGATION_NONE }, + { "policy", CURLGSSAPI_DELEGATION_POLICY_FLAG }, + { "always", CURLGSSAPI_DELEGATION_FLAG }, +}; +#endif + static struct credential proxy_auth = CREDENTIAL_INIT; static const char *curl_proxyuserpwd; static const char *curl_cookie_file; @@@ -111,6 -98,8 +111,8 @@@ static int http_proactive_auth static const char *user_agent; static int curl_empty_auth; + enum http_follow_config http_follow_config = HTTP_FOLLOW_INITIAL; + #if LIBCURL_VERSION_NUM >= 0x071700 /* Use CURLOPT_KEYPASSWD as is */ #elif LIBCURL_VERSION_NUM >= 0x070903 @@@ -213,13 -202,6 +215,13 @@@ static void finish_active_slot(struct a slot->callback_func(slot->callback_data); } +static void xmulti_remove_handle(struct active_request_slot *slot) +{ +#ifdef USE_CURL_MULTI + curl_multi_remove_handle(curlm, slot->curl); +#endif +} + #ifdef USE_CURL_MULTI static void process_curl_messages(void) { @@@ -235,7 -217,7 +237,7 @@@ slot->curl != curl_message->easy_handle) slot = slot->next; if (slot != NULL) { - curl_multi_remove_handle(curlm, slot->curl); + xmulti_remove_handle(slot); slot->curl_result = curl_result; finish_active_slot(slot); } else { @@@ -335,15 -317,6 +337,15 @@@ static int http_options(const char *var return 0; } + if (!strcmp("http.delegation", var)) { +#if LIBCURL_VERSION_NUM >= 0x071600 + return git_config_string(&curl_deleg, var, value); +#else + warning(_("Delegation control is not supported with cURL < 7.22.0")); + return 0; +#endif + } + if (!strcmp("http.pinnedpubkey", var)) { #if LIBCURL_VERSION_NUM >= 0x072c00 return git_config_pathname(&ssl_pinnedkey, var, value); @@@ -366,13 -339,23 +368,23 @@@ return 0; } + if (!strcmp("http.followredirects", var)) { + if (value && !strcmp(value, "initial")) + http_follow_config = HTTP_FOLLOW_INITIAL; + else if (git_config_bool(var, value)) + http_follow_config = HTTP_FOLLOW_ALWAYS; + else + http_follow_config = HTTP_FOLLOW_NONE; + return 0; + } + /* Fall back on the default ones */ return git_default_config(var, value, cb); } static void init_curl_http_auth(CURL *result) { - if (!http_auth.username) { + if (!http_auth.username || !*http_auth.username) { if (curl_empty_auth) curl_easy_setopt(result, CURLOPT_USERPWD, ":"); return; @@@ -506,125 -489,6 +518,125 @@@ static void set_curl_keepalive(CURL *c } #endif +static void redact_sensitive_header(struct strbuf *header) +{ + const char *sensitive_header; + + if (skip_prefix(header->buf, "Authorization:", &sensitive_header) || + skip_prefix(header->buf, "Proxy-Authorization:", &sensitive_header)) { + /* The first token is the type, which is OK to log */ + while (isspace(*sensitive_header)) + sensitive_header++; + while (*sensitive_header && !isspace(*sensitive_header)) + sensitive_header++; + /* Everything else is opaque and possibly sensitive */ + strbuf_setlen(header, sensitive_header - header->buf); + strbuf_addstr(header, " "); + } +} + +static void curl_dump_header(const char *text, unsigned char *ptr, size_t size, int hide_sensitive_header) +{ + struct strbuf out = STRBUF_INIT; + struct strbuf **headers, **header; + + strbuf_addf(&out, "%s, %10.10ld bytes (0x%8.8lx)\n", + text, (long)size, (long)size); + trace_strbuf(&trace_curl, &out); + strbuf_reset(&out); + strbuf_add(&out, ptr, size); + headers = strbuf_split_max(&out, '\n', 0); + + for (header = headers; *header; header++) { + if (hide_sensitive_header) + redact_sensitive_header(*header); + strbuf_insert((*header), 0, text, strlen(text)); + strbuf_insert((*header), strlen(text), ": ", 2); + strbuf_rtrim((*header)); + strbuf_addch((*header), '\n'); + trace_strbuf(&trace_curl, (*header)); + } + strbuf_list_free(headers); + strbuf_release(&out); +} + +static void curl_dump_data(const char *text, unsigned char *ptr, size_t size) +{ + size_t i; + struct strbuf out = STRBUF_INIT; + unsigned int width = 60; + + strbuf_addf(&out, "%s, %10.10ld bytes (0x%8.8lx)\n", + text, (long)size, (long)size); + trace_strbuf(&trace_curl, &out); + + for (i = 0; i < size; i += width) { + size_t w; + + strbuf_reset(&out); + strbuf_addf(&out, "%s: ", text); + for (w = 0; (w < width) && (i + w < size); w++) { + unsigned char ch = ptr[i + w]; + + strbuf_addch(&out, + (ch >= 0x20) && (ch < 0x80) + ? ch : '.'); + } + strbuf_addch(&out, '\n'); + trace_strbuf(&trace_curl, &out); + } + strbuf_release(&out); +} + +static int curl_trace(CURL *handle, curl_infotype type, char *data, size_t size, void *userp) +{ + const char *text; + enum { NO_FILTER = 0, DO_FILTER = 1 }; + + switch (type) { + case CURLINFO_TEXT: + trace_printf_key(&trace_curl, "== Info: %s", data); + default: /* we ignore unknown types by default */ + return 0; + + case CURLINFO_HEADER_OUT: + text = "=> Send header"; + curl_dump_header(text, (unsigned char *)data, size, DO_FILTER); + break; + case CURLINFO_DATA_OUT: + text = "=> Send data"; + curl_dump_data(text, (unsigned char *)data, size); + break; + case CURLINFO_SSL_DATA_OUT: + text = "=> Send SSL data"; + curl_dump_data(text, (unsigned char *)data, size); + break; + case CURLINFO_HEADER_IN: + text = "<= Recv header"; + curl_dump_header(text, (unsigned char *)data, size, NO_FILTER); + break; + case CURLINFO_DATA_IN: + text = "<= Recv data"; + curl_dump_data(text, (unsigned char *)data, size); + break; + case CURLINFO_SSL_DATA_IN: + text = "<= Recv SSL data"; + curl_dump_data(text, (unsigned char *)data, size); + break; + } + return 0; +} + +void setup_curl_trace(CURL *handle) +{ + if (!trace_want(&trace_curl)) + return; + curl_easy_setopt(handle, CURLOPT_VERBOSE, 1L); + curl_easy_setopt(handle, CURLOPT_DEBUGFUNCTION, curl_trace); + curl_easy_setopt(handle, CURLOPT_DEBUGDATA, NULL); +} + + static CURL *get_curl_handle(void) { CURL *result = curl_easy_init(); @@@ -650,22 -514,6 +662,22 @@@ curl_easy_setopt(result, CURLOPT_HTTPAUTH, CURLAUTH_ANY); #endif +#if LIBCURL_VERSION_NUM >= 0x071600 + if (curl_deleg) { + int i; + for (i = 0; i < ARRAY_SIZE(curl_deleg_levels); i++) { + if (!strcmp(curl_deleg, curl_deleg_levels[i].name)) { + curl_easy_setopt(result, CURLOPT_GSSAPI_DELEGATION, + curl_deleg_levels[i].curl_deleg_param); + break; + } + } + if (i == ARRAY_SIZE(curl_deleg_levels)) + warning("Unknown delegation method '%s': using default", + curl_deleg); + } +#endif + if (http_proactive_auth) init_curl_http_auth(result); @@@ -717,7 -565,6 +729,6 @@@ curl_low_speed_time); } - curl_easy_setopt(result, CURLOPT_FOLLOWLOCATION, 1); curl_easy_setopt(result, CURLOPT_MAXREDIRS, 20); #if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_POSTREDIR, CURL_REDIR_POST_ALL); @@@ -734,14 -581,15 +745,15 @@@ if (is_transport_allowed("ftps")) allowed_protocols |= CURLPROTO_FTPS; curl_easy_setopt(result, CURLOPT_REDIR_PROTOCOLS, allowed_protocols); + curl_easy_setopt(result, CURLOPT_PROTOCOLS, allowed_protocols); #else if (transport_restrict_protocols()) warning("protocol restrictions not applied to curl redirects because\n" "your curl version is too old (>= 7.19.4)"); #endif - if (getenv("GIT_CURL_VERBOSE")) - curl_easy_setopt(result, CURLOPT_VERBOSE, 1); + curl_easy_setopt(result, CURLOPT_VERBOSE, 1L); + setup_curl_trace(result); curl_easy_setopt(result, CURLOPT_USERAGENT, user_agent ? user_agent : git_user_agent()); @@@ -767,7 -615,7 +779,7 @@@ * precedence here, as in CURL. */ if (!curl_http_proxy) { - if (!strcmp(http_auth.protocol, "https")) { + if (http_auth.protocol && !strcmp(http_auth.protocol, "https")) { var_override(&curl_http_proxy, getenv("HTTPS_PROXY")); var_override(&curl_http_proxy, getenv("https_proxy")); } else { @@@ -925,7 -773,9 +937,7 @@@ void http_cleanup(void while (slot != NULL) { struct active_request_slot *next = slot->next; if (slot->curl != NULL) { -#ifdef USE_CURL_MULTI - curl_multi_remove_handle(curlm, slot->curl); -#endif + xmulti_remove_handle(slot); curl_easy_cleanup(slot->curl); } free(slot); @@@ -1044,6 -894,16 +1056,16 @@@ struct active_request_slot *get_active_ curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 1); curl_easy_setopt(slot->curl, CURLOPT_RANGE, NULL); + /* + * Default following to off unless "ALWAYS" is configured; this gives + * callers a sane starting point, and they can tweak for individual + * HTTP_FOLLOW_* cases themselves. + */ + if (http_follow_config == HTTP_FOLLOW_ALWAYS) + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1); + else + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 0); + #if LIBCURL_VERSION_NUM >= 0x070a08 curl_easy_setopt(slot->curl, CURLOPT_IPRESOLVE, git_curl_ipresolve); #endif @@@ -1064,8 -924,6 +1086,8 @@@ int start_active_slot(struct active_req if (curlm_result != CURLM_OK && curlm_result != CURLM_CALL_MULTI_PERFORM) { + warning("curl_multi_add_handle failed: %s", + curl_multi_strerror(curlm_result)); active_requests--; slot->in_use = 0; return 0; @@@ -1205,13 -1063,13 +1227,13 @@@ void run_active_slot(struct active_requ static void release_active_slot(struct active_request_slot *slot) { closedown_active_slot(slot); - if (slot->curl && curl_session_count > min_curl_sessions) { -#ifdef USE_CURL_MULTI - curl_multi_remove_handle(curlm, slot->curl); -#endif - curl_easy_cleanup(slot->curl); - slot->curl = NULL; - curl_session_count--; + if (slot->curl) { + xmulti_remove_handle(slot); + if (curl_session_count > min_curl_sessions) { + curl_easy_cleanup(slot->curl); + slot->curl = NULL; + curl_session_count--; + } } #ifdef USE_CURL_MULTI fill_active_slots(); @@@ -1286,9 -1144,12 +1308,12 @@@ static int handle_curl_result(struct sl * If we see a failing http code with CURLE_OK, we have turned off * FAILONERROR (to keep the server's custom error response), and should * translate the code into failure here. + * + * Likewise, if we see a redirect (30x code), that means we turned off + * redirect-following, and we should treat the result as an error. */ if (results->curl_result == CURLE_OK && - results->http_code >= 400) { + results->http_code >= 300) { results->curl_result = CURLE_HTTP_RETURNED_ERROR; /* * Normally curl will already have put the "reason phrase" @@@ -1607,6 -1468,9 +1632,9 @@@ static int http_request(const char *url strbuf_addstr(&buf, " no-cache"); if (options && options->keep_error) curl_easy_setopt(slot->curl, CURLOPT_FAILONERROR, 0); + if (options && options->initial_request && + http_follow_config == HTTP_FOLLOW_INITIAL) + curl_easy_setopt(slot->curl, CURLOPT_FOLLOWLOCATION, 1); headers = curl_slist_append(headers, buf.buf); @@@ -1655,16 -1519,16 +1683,16 @@@ * * Note that this assumes a sane redirect scheme. It's entirely possible * in the example above to end up at a URL that does not even end in - * "info/refs". In such a case we simply punt, as there is not much we can - * do (and such a scheme is unlikely to represent a real git repository, - * which means we are likely about to abort anyway). + * "info/refs". In such a case we die. There's not much we can do, such a + * scheme is unlikely to represent a real git repository, and failing to + * rewrite the base opens options for malicious redirects to do funny things. */ static int update_url_from_redirect(struct strbuf *base, const char *asked, const struct strbuf *got) { const char *tail; - size_t tail_len; + size_t new_len; if (!strcmp(asked, got->buf)) return 0; @@@ -1673,14 -1537,16 +1701,16 @@@ die("BUG: update_url_from_redirect: %s is not a superset of %s", asked, base->buf); - tail_len = strlen(tail); - - if (got->len < tail_len || - strcmp(tail, got->buf + got->len - tail_len)) - return 0; /* insane redirect scheme */ + new_len = got->len; + if (!strip_suffix_mem(got->buf, &new_len, tail)) + die(_("unable to update url base from redirection:\n" + " asked for: %s\n" + " redirect: %s"), + asked, got->buf); strbuf_reset(base); - strbuf_add(base, got->buf, got->len - tail_len); + strbuf_add(base, got->buf, new_len); + return 1; } @@@ -2028,7 -1894,7 +2058,7 @@@ static size_t fwrite_sha1_file(char *pt if (c != CURLE_OK) die("BUG: curl_easy_getinfo for HTTP code failed: %s", curl_easy_strerror(c)); - if (slot->http_code >= 400) + if (slot->http_code >= 300) return size; } diff --combined http.h index 5ab9d9c329,31b4cc94be..02bccb7b0c --- a/http.h +++ b/http.h @@@ -116,6 -116,13 +116,13 @@@ extern struct credential http_auth extern char curl_errorstr[CURL_ERROR_SIZE]; + enum http_follow_config { + HTTP_FOLLOW_NONE, + HTTP_FOLLOW_ALWAYS, + HTTP_FOLLOW_INITIAL + }; + extern enum http_follow_config http_follow_config; + static inline int missing__target(int code, int result) { return /* file:// URL -- do we ever use one??? */ @@@ -139,7 -146,8 +146,8 @@@ extern char *get_remote_object_url(cons /* Options for http_get_*() */ struct http_get_options { unsigned no_cache:1, - keep_error:1; + keep_error:1, + initial_request:1; /* If non-NULL, returns the content-type of the response. */ struct strbuf *content_type; @@@ -225,6 -233,4 +233,6 @@@ extern int finish_http_object_request(s extern void abort_http_object_request(struct http_object_request *freq); extern void release_http_object_request(struct http_object_request *freq); +/* setup routine for curl_easy_setopt CURLOPT_DEBUGFUNCTION */ +void setup_curl_trace(CURL *handle); #endif /* HTTP_H */ diff --combined remote-curl.c index f14c41f4c0,05ae8dead7..28d9d10638 --- a/remote-curl.c +++ b/remote-curl.c @@@ -20,8 -20,6 +20,8 @@@ static struct strbuf url = STRBUF_INIT struct options { int verbosity; unsigned long depth; + char *deepen_since; + struct string_list deepen_not; unsigned progress : 1, check_self_contained_and_connected : 1, cloning : 1, @@@ -30,8 -28,7 +30,8 @@@ dry_run : 1, thin : 1, /* One of the SEND_PACK_PUSH_CERT_* constants. */ - push_cert : 2; + push_cert : 2, + deepen_relative : 1; }; static struct options options; static struct string_list cas_options = STRING_LIST_INIT_DUP; @@@ -63,23 -60,6 +63,23 @@@ static int set_option(const char *name options.depth = v; return 0; } + else if (!strcmp(name, "deepen-since")) { + options.deepen_since = xstrdup(value); + return 0; + } + else if (!strcmp(name, "deepen-not")) { + string_list_append(&options.deepen_not, value); + return 0; + } + else if (!strcmp(name, "deepen-relative")) { + if (!strcmp(value, "true")) + options.deepen_relative = 1; + else if (!strcmp(value, "false")) + options.deepen_relative = 0; + else + return -1; + return 0; + } else if (!strcmp(name, "followtags")) { if (!strcmp(value, "true")) options.followtags = 1; @@@ -274,7 -254,7 +274,7 @@@ static struct discovery *discover_refs( struct strbuf effective_url = STRBUF_INIT; struct discovery *last = last_discovery; int http_ret, maybe_smart = 0; - struct http_get_options options; + struct http_get_options http_options; if (last && !strcmp(service, last->service)) return last; @@@ -291,15 -271,16 +291,16 @@@ strbuf_addf(&refs_url, "service=%s", service); } - memset(&options, 0, sizeof(options)); - options.content_type = &type; - options.charset = &charset; - options.effective_url = &effective_url; - options.base_url = &url; - options.no_cache = 1; - options.keep_error = 1; + memset(&http_options, 0, sizeof(http_options)); + http_options.content_type = &type; + http_options.charset = &charset; + http_options.effective_url = &effective_url; + http_options.base_url = &url; + http_options.initial_request = 1; + http_options.no_cache = 1; + http_options.keep_error = 1; - http_ret = http_get_strbuf(refs_url.buf, &buffer, &options); + http_ret = http_get_strbuf(refs_url.buf, &buffer, &http_options); switch (http_ret) { case HTTP_OK: break; @@@ -314,6 -295,9 +315,9 @@@ die("unable to access '%s': %s", url.buf, curl_errorstr); } + if (options.verbosity && !starts_with(refs_url.buf, url.buf)) + warning(_("redirecting to %s"), url.buf); + last= xcalloc(1, sizeof(*last_discovery)); last->service = service; last->buf_alloc = strbuf_detach(&buffer, &last->len); @@@ -745,8 -729,8 +749,8 @@@ static int fetch_dumb(int nr_heads, str int ret, i; ALLOC_ARRAY(targets, nr_heads); - if (options.depth) - die("dumb http transport does not support --depth"); + if (options.depth || options.deepen_since) + die("dumb http transport does not support shallow capabilities"); for (i = 0; i < nr_heads; i++) targets[i] = xstrdup(oid_to_hex(&to_fetch[i]->old_oid)); @@@ -771,35 -755,38 +775,35 @@@ static int fetch_git(struct discovery * { struct rpc_state rpc; struct strbuf preamble = STRBUF_INIT; - char *depth_arg = NULL; - int argc = 0, i, err; - const char *argv[17]; - - argv[argc++] = "fetch-pack"; - argv[argc++] = "--stateless-rpc"; - argv[argc++] = "--stdin"; - argv[argc++] = "--lock-pack"; + int i, err; + struct argv_array args = ARGV_ARRAY_INIT; + + argv_array_pushl(&args, "fetch-pack", "--stateless-rpc", + "--stdin", "--lock-pack", NULL); if (options.followtags) - argv[argc++] = "--include-tag"; + argv_array_push(&args, "--include-tag"); if (options.thin) - argv[argc++] = "--thin"; - if (options.verbosity >= 3) { - argv[argc++] = "-v"; - argv[argc++] = "-v"; - } + argv_array_push(&args, "--thin"); + if (options.verbosity >= 3) + argv_array_pushl(&args, "-v", "-v", NULL); if (options.check_self_contained_and_connected) - argv[argc++] = "--check-self-contained-and-connected"; + argv_array_push(&args, "--check-self-contained-and-connected"); if (options.cloning) - argv[argc++] = "--cloning"; + argv_array_push(&args, "--cloning"); if (options.update_shallow) - argv[argc++] = "--update-shallow"; + argv_array_push(&args, "--update-shallow"); if (!options.progress) - argv[argc++] = "--no-progress"; - if (options.depth) { - struct strbuf buf = STRBUF_INIT; - strbuf_addf(&buf, "--depth=%lu", options.depth); - depth_arg = strbuf_detach(&buf, NULL); - argv[argc++] = depth_arg; - } - argv[argc++] = url.buf; - argv[argc++] = NULL; + argv_array_push(&args, "--no-progress"); + if (options.depth) + argv_array_pushf(&args, "--depth=%lu", options.depth); + if (options.deepen_since) + argv_array_pushf(&args, "--shallow-since=%s", options.deepen_since); + for (i = 0; i < options.deepen_not.nr; i++) + argv_array_pushf(&args, "--shallow-exclude=%s", + options.deepen_not.items[i].string); + if (options.deepen_relative && options.depth) + argv_array_push(&args, "--deepen-relative"); + argv_array_push(&args, url.buf); for (i = 0; i < nr_heads; i++) { struct ref *ref = to_fetch[i]; @@@ -812,7 -799,7 +816,7 @@@ memset(&rpc, 0, sizeof(rpc)); rpc.service_name = "git-upload-pack", - rpc.argv = argv; + rpc.argv = args.argv; rpc.stdin_preamble = &preamble; rpc.gzip_request = 1; @@@ -821,7 -808,7 +825,7 @@@ write_or_die(1, rpc.result.buf, rpc.result.len); strbuf_release(&rpc.result); strbuf_release(&preamble); - free(depth_arg); + argv_array_clear(&args); return err; } @@@ -1015,7 -1002,6 +1019,7 @@@ int cmd_main(int argc, const char **arg options.verbosity = 1; options.progress = !!isatty(2); options.thin = 1; + string_list_init(&options.deepen_not, 1); remote = remote_get(argv[1]); diff --combined t/lib-httpd/apache.conf index c3e631394f,5b408d6495..69174c6e31 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@@ -123,6 -123,7 +123,7 @@@ ScriptAlias /error/ error.sh RewriteEngine on + RewriteRule ^/dumb-redir/(.*)$ /dumb/$1 [R=301] RewriteRule ^/smart-redir-perm/(.*)$ /smart/$1 [R=301] RewriteRule ^/smart-redir-temp/(.*)$ /smart/$1 [R=302] RewriteRule ^/smart-redir-auth/(.*)$ /auth/smart/$1 [R=301] @@@ -132,6 -133,19 +133,19 @@@ RewriteRule ^/ftp-redir/(.*)$ ftp://loc RewriteRule ^/loop-redir/x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-x-(.*) /$1 [R=302] RewriteRule ^/loop-redir/(.*)$ /loop-redir/x-$1 [R=302] + # The first rule issues a client-side redirect to something + # that _doesn't_ look like a git repo. The second rule is a + # server-side rewrite, so that it turns out the odd-looking + # thing _is_ a git repo. The "[PT]" tells Apache to match + # the usual ScriptAlias rules for /smart. + RewriteRule ^/insane-redir/(.*)$ /intern-redir/$1/foo [R=301] + RewriteRule ^/intern-redir/(.*)/foo$ /smart/$1 [PT] + + # Serve info/refs internally without redirecting, but + # issue a redirect for any object requests. + RewriteRule ^/redir-objects/(.*/info/refs)$ /dumb/$1 [PT] + RewriteRule ^/redir-objects/(.*/objects/.*)$ /dumb/$1 [R=301] + # Apache 2.2 does not understand , so we use RewriteCond. # And as RewriteCond does not allow testing for non-matches, we match # the desired case first (one has abra, two has cadabra), and let it @@@ -208,8 -222,8 +222,8 @@@ RewriteRule ^/half-auth-complete/ - [E= LoadModule dav_svn_module modules/mod_dav_svn.so - + DAV svn - SVNPath svnrepo + SVNPath "${LIB_HTTPD_SVNPATH}" diff --combined t/t5550-http-fetch-dumb.sh index 7641417b4a,22011f0b68..264a1ab8b0 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@@ -263,15 -263,15 +263,15 @@@ check_language () >expect ;; ?*) - echo "Accept-Language: $1" >expect + echo "=> Send header: Accept-Language: $1" >expect ;; esac && - GIT_CURL_VERBOSE=1 \ + GIT_TRACE_CURL=true \ LANGUAGE=$2 \ git ls-remote "$HTTPD_URL/dumb/repo.git" >output 2>&1 && tr -d '\015' actual && + sed -ne '/^=> Send header: Accept-Language:/ p' >actual && test_cmp expect actual } @@@ -295,17 -295,70 +295,78 @@@ ja;q=0.95, zh;q=0.94, sv;q=0.93, pt;q=0 ' test_expect_success 'git client does not send an empty Accept-Language' ' - GIT_CURL_VERBOSE=1 LANGUAGE= git ls-remote "$HTTPD_URL/dumb/repo.git" 2>stderr && - ! grep "^Accept-Language:" stderr + GIT_TRACE_CURL=true LANGUAGE= git ls-remote "$HTTPD_URL/dumb/repo.git" 2>stderr && + ! grep "^=> Send header: Accept-Language:" stderr +' + +test_expect_success 'remote-http complains cleanly about malformed urls' ' + # do not actually issue "list" or other commands, as we do not + # want to rely on what curl would actually do with such a broken + # URL. This is just about making sure we do not segfault during + # initialization. + test_must_fail git remote-http http::/example.com/repo.git ' + test_expect_success 'redirects can be forbidden/allowed' ' + test_must_fail git -c http.followRedirects=false \ + clone $HTTPD_URL/dumb-redir/repo.git dumb-redir && + git -c http.followRedirects=true \ + clone $HTTPD_URL/dumb-redir/repo.git dumb-redir 2>stderr + ' + + test_expect_success 'redirects are reported to stderr' ' + # just look for a snippet of the redirected-to URL + test_i18ngrep /dumb/ stderr + ' + + test_expect_success 'non-initial redirects can be forbidden' ' + test_must_fail git -c http.followRedirects=initial \ + clone $HTTPD_URL/redir-objects/repo.git redir-objects && + git -c http.followRedirects=true \ + clone $HTTPD_URL/redir-objects/repo.git redir-objects + ' + + test_expect_success 'http.followRedirects defaults to "initial"' ' + test_must_fail git clone $HTTPD_URL/redir-objects/repo.git default + ' + + # The goal is for a clone of the "evil" repository, which has no objects + # itself, to cause the client to fetch objects from the "victim" repository. + test_expect_success 'set up evil alternates scheme' ' + victim=$HTTPD_DOCUMENT_ROOT_PATH/victim.git && + git init --bare "$victim" && + git -C "$victim" --work-tree=. commit --allow-empty -m secret && + git -C "$victim" repack -ad && + git -C "$victim" update-server-info && + sha1=$(git -C "$victim" rev-parse HEAD) && + + evil=$HTTPD_DOCUMENT_ROOT_PATH/evil.git && + git init --bare "$evil" && + # do this by hand to avoid object existence check + printf "%s\\t%s\\n" $sha1 refs/heads/master >"$evil/info/refs" + ' + + # Here we'll just redirect via HTTP. In a real-world attack these would be on + # different servers, but we should reject it either way. + test_expect_success 'http-alternates is a non-initial redirect' ' + echo "$HTTPD_URL/dumb/victim.git/objects" \ + >"$evil/objects/info/http-alternates" && + test_must_fail git -c http.followRedirects=initial \ + clone $HTTPD_URL/dumb/evil.git evil-initial && + git -c http.followRedirects=true \ + clone $HTTPD_URL/dumb/evil.git evil-initial + ' + + # Curl supports a lot of protocols that we'd prefer not to allow + # http-alternates to use, but it's hard to test whether curl has + # accessed, say, the SMTP protocol, because we are not running an SMTP server. + # But we can check that it does not allow access to file://, which would + # otherwise allow this clone to complete. + test_expect_success 'http-alternates cannot point at funny protocols' ' + echo "file://$victim/objects" >"$evil/objects/info/http-alternates" && + test_must_fail git -c http.followRedirects=true \ + clone "$HTTPD_URL/dumb/evil.git" evil-file + ' + stop_httpd test_done diff --combined t/t5551-http-fetch-smart.sh index 1ec5b2747a,d8826acde6..6e5b9e42fb --- a/t/t5551-http-fetch-smart.sh +++ b/t/t5551-http-fetch-smart.sh @@@ -43,21 -43,12 +43,21 @@@ cat >exp <err && + GIT_TRACE_CURL=true git clone --quiet $HTTPD_URL/smart/repo.git clone 2>err && test_cmp file clone/file && tr '\''\015'\'' Q Send header, /d + /^=> Send header:$/d + /^<= Recv header, /d + /^<= Recv header:$/d + s/=> Send header: // + s/= Recv header:// + /^<= Recv data/d + /^=> Send data/d /^$/d /^< $/d @@@ -119,6 -110,10 +119,10 @@@ test_expect_success 'redirects re-root git clone $HTTPD_URL/smart-redir-limited/repo.git repo-redir-limited ' + test_expect_success 're-rooting dies on insane schemes' ' + test_must_fail git clone $HTTPD_URL/insane-redir/repo.git insane + ' + test_expect_success 'clone from password-protected repository' ' echo two >expect && set_askpass user@host pass@host && @@@ -270,9 -265,9 +274,9 @@@ test_expect_success CMDLINE_LIMIT ' test_expect_success 'large fetch-pack requests can be split across POSTs' ' - GIT_CURL_VERBOSE=1 git -c http.postbuffer=65536 \ + GIT_TRACE_CURL=true git -c http.postbuffer=65536 \ clone --bare "$HTTPD_URL/smart/repo.git" split.git 2>err && - grep "^> POST" err >posts && + grep "^=> Send header: POST" err >posts && test_line_count = 2 posts '