Support RFC 2822 date parsing in fast-import.

Since some frontends may be working with source material where
the dates are only readily available as RFC 2822 strings, it is
more friendly if fast-import exposes Git's parse_date() function
to handle the conversion.  This way the frontend doesn't need
to perform the parsing itself.

The new --date-format option to fast-import can be used by a
frontend to select which format it will supply date strings in.
The default is the standard `raw` Git format, which fast-import
has always supported.  Format rfc2822 can be used to activate the
parse_date() function instead.

Because fast-import could also be useful for creating new, current
commits, the format `now` is also supported to generate the current
system timestamp.  The implementation of `now` is a trivial call
to datestamp(), but is actually a whole whopping 3 lines so that
fast-import can verify the frontend really meant `now`.

As part of this change I have added validation of the `raw` date
format.  Prior to this change fast-import would accept anything
in a `committer` command, even if it was seriously malformed.
Now fast-import requires the '> ' near the end of the string and
verifies the timestamp is formatted properly.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

diff --git a/Documentation/git-fast-import.txt b/Documentation/git-fast-import.txt
index 6fc78bff3e5aa08a3dc133cd356966d5aba86724..08450de9ac0e9ba7c116a9c1c92b806458073541 100644 (file)
--- a/Documentation/git-fast-import.txt
+++ b/Documentation/git-fast-import.txt
@@ -32,6 +32,12 @@ the frontend program in use.
 
 OPTIONS
 -------
 
 OPTIONS
 -------

+--date-format=<fmt>::
+       Specify the type of dates the frontend will supply to
+       gfi within `author`, `committer` and `tagger` commands.
+       See ``Date Formats'' below for details about which formats
+       are supported, and their syntax.
+

 --max-pack-size=<n>::
        Maximum size of each output packfile, expressed in MiB.
        The default is 4096 (4 GiB) as that is the maximum allowed
 --max-pack-size=<n>::
        Maximum size of each output packfile, expressed in MiB.
        The default is 4096 (4 GiB) as that is the maximum allowed


@@ -53,7 +59,6 @@ OPTIONS
        Frontends can use this file to validate imports after they
        have been completed.
 
        Frontends can use this file to validate imports after they
        have been completed.
 

-

 Performance
 -----------
 The design of gfi allows it to import large projects in a minimum
 Performance
 -----------
 The design of gfi allows it to import large projects in a minimum


@@ -127,6 +132,78 @@ results, such as branch names or file names with leading or trailing
 spaces in their name, or early termination of gfi when it encounters
 unexpected input.
 
 spaces in their name, or early termination of gfi when it encounters
 unexpected input.
 

+Date Formats
+~~~~~~~~~~~~
+The following date formats are supported.  A frontend should select
+the format it will use for this import by passing the format name
+in the `--date-format=<fmt>` command line option.
+
+`raw`::
+       This is the Git native format and is `<time> SP <tz>`.
+       It is also gfi's default format, if `--date-format` was
+       not specified.
++
+The time of the event is specified by `<time>` as the number of
+seconds since the UNIX epoch (midnight, Jan 1, 1970, UTC) and is
+written as an ASCII decimal integer.
++
+The timezone is specified by `<tz>` as a positive or negative offset
+from UTC.  For example EST (which is typically 5 hours behind GMT)
+would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
++
+If the timezone is not available in the source material, use
+``+0000'', or the most common local timezone.  For example many
+organizations have a CVS repository which has only ever been accessed
+by users who are located in the same location and timezone.  In this
+case the user's timezone can be easily assumed.
++
+Unlike the `rfc2822` format, this format is very strict.  Any
+variation in formatting will cause gfi to reject the value.
+
+`rfc2822`::
+       This is the standard email format as described by RFC 2822.
++
+An example value is ``Tue Feb 6 11:22:18 2007 -0500''.  The Git
+parser is accurate, but a little on the lenient side.  Its the
+same parser used by gitlink:git-am[1] when applying patches
+received from email.
++
+Some malformed strings may be accepted as valid dates.  In some of
+these cases Git will still be able to obtain the correct date from
+the malformed string.  There are also some types of malformed
+strings which Git will parse wrong, and yet consider valid.
+Seriously malformed strings will be rejected.
++
+If the source material is formatted in RFC 2822 style dates,
+the frontend should let gfi handle the parsing and conversion
+(rather than attempting to do it itself) as the Git parser has
+been well tested in the wild.
++
+Frontends should prefer the `raw` format if the source material
+is already in UNIX-epoch format, or is easily convertible to
+that format, as there is no ambiguity in parsing.
+
+`now`::
+       Always use the current time and timezone.  The literal
+       `now` must always be supplied for `<when>`.
++
+This is a toy format.  The current time and timezone of this system
+is always copied into the identity string at the time it is being
+created by gfi.  There is no way to specify a different time or
+timezone.
++
+This particular format is supplied as its short to implement and
+may be useful to a process that wants to create a new commit
+right now, without needing to use a working directory or
+gitlink:git-update-index[1].
++
+If separate `author` and `committer` commands are used in a `commit`
+the timestamps may not match, as the system clock will be polled
+twice (once for each command).  The only way to ensure that both
+author and committer identity information has the same timestamp
+is to omit `author` (thus copying from `committer`) or to use a
+date format other than `now`.
+

 Commands
 ~~~~~~~~
 gfi accepts several commands to update the current repository
 Commands
 ~~~~~~~~
 gfi accepts several commands to update the current repository


@@ -168,8 +245,8 @@ change to the project.
 ....
        'commit' SP <ref> LF
        mark?
 ....
        'commit' SP <ref> LF
        mark?

-       ('author' SP <name> SP LT <email> GT SP <time> SP <tz> LF)?
-       'committer' SP <name> SP LT <email> GT SP <time> SP <tz> LF
+       ('author' SP <name> SP LT <email> GT SP <when> LF)?
+       'committer' SP <name> SP LT <email> GT SP <when> LF

        data
        ('from' SP <committish> LF)?
        ('merge' SP <committish> LF)?
        data
        ('from' SP <committish> LF)?
        ('merge' SP <committish> LF)?


@@ -222,12 +299,10 @@ the email address from the other fields in the line.  Note that
 `<name>` is free-form and may contain any sequence of bytes, except
 `LT` and `LF`.  It is typically UTF-8 encoded.
 
 `<name>` is free-form and may contain any sequence of bytes, except
 `LT` and `LF`.  It is typically UTF-8 encoded.
 

-The time of the change is specified by `<time>` as the number of
-seconds since the UNIX epoc (midnight, Jan 1, 1970, UTC) and is
-written as an ASCII decimal integer.  The committer's
-timezone is specified by `<tz>` as a positive or negative offset
-from UTC.  For example EST (which is typically 5 hours behind GMT)
-would be expressed in `<tz>` by ``-0500'' while GMT is ``+0000''.
+The time of the change is specified by `<when>` using the date format
+that was selected by the `--date-format=<fmt>` command line option.
+See ``Date Formats'' above for the set of supported formats, and
+their syntax.

 
 `from`
 ^^^^^^
 
 `from`
 ^^^^^^


@@ -394,7 +469,7 @@ lightweight (non-annotated) tags see the `reset` command below.
 ....
        'tag' SP <name> LF
        'from' SP <committish> LF
 ....
        'tag' SP <name> LF
        'from' SP <committish> LF

-       'tagger' SP <name> SP LT <email> GT SP <time> SP <tz> LF
+       'tagger' SP <name> SP LT <email> GT SP <when> LF

        data
        LF
 ....
        data
        LF
 ....

diff --git a/fast-import.c b/fast-import.c
index 4dcba416e098efc785fe7b28a37f1fa823d5548b..ee4777fdafcbedc24997155581e55c339b40981e 100644 (file)
--- a/fast-import.c
+++ b/fast-import.c
@@ -17,8 +17,8 @@ Format of STDIN stream:
 
   new_commit ::= 'commit' sp ref_str lf
     mark?
 
   new_commit ::= 'commit' sp ref_str lf
     mark?

-    ('author' sp name '<' email '>' ts tz lf)?
-    'committer' sp name '<' email '>' ts tz lf
+    ('author' sp name '<' email '>' when lf)?
+    'committer' sp name '<' email '>' when lf

     commit_msg
     ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?
     ('merge' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)*
     commit_msg
     ('from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)?
     ('merge' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf)*


@@ -34,7 +34,7 @@ Format of STDIN stream:
 
   new_tag ::= 'tag' sp tag_str lf
     'from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf
 
   new_tag ::= 'tag' sp tag_str lf
     'from' sp (ref_str | hexsha1 | sha1exp_str | idnum) lf

-       'tagger' sp name '<' email '>' ts tz lf
+       'tagger' sp name '<' email '>' when lf

     tag_msg;
   tag_msg ::= data;
 
     tag_msg;
   tag_msg ::= data;
 


@@ -88,6 +88,10 @@ Format of STDIN stream:
   bigint ::= # unsigned integer value, ascii base10 notation;
   binary_data ::= # file content, not interpreted;
 
   bigint ::= # unsigned integer value, ascii base10 notation;
   binary_data ::= # file content, not interpreted;
 

+  when         ::= raw_when | rfc2822_when;
+  raw_when     ::= ts sp tz;
+  rfc2822_when ::= # Valid RFC 2822 date and time;
+

   sp ::= # ASCII space character;
   lf ::= # ASCII newline (LF) character;
 
   sp ::= # ASCII space character;
   lf ::= # ASCII newline (LF) character;
 


@@ -234,6 +238,12 @@ struct hash_list
        unsigned char sha1[20];
 };
 
        unsigned char sha1[20];
 };
 

+typedef enum {
+       WHENSPEC_RAW = 1,
+       WHENSPEC_RFC2822,
+       WHENSPEC_NOW,
+} whenspec_type;
+

 /* Configured limits on output */
 static unsigned long max_depth = 10;
 static unsigned long max_packsize = (1LL << 32) - 1;
 /* Configured limits on output */
 static unsigned long max_depth = 10;
 static unsigned long max_packsize = (1LL << 32) - 1;


@@ -294,6 +304,7 @@ static struct tag *first_tag;
 static struct tag *last_tag;
 
 /* Input stream parsing */
 static struct tag *last_tag;
 
 /* Input stream parsing */

+static whenspec_type whenspec = WHENSPEC_RAW;

 static struct strbuf command_buf;
 static uintmax_t next_mark;
 static struct dbuf new_data;
 static struct strbuf command_buf;
 static uintmax_t next_mark;
 static struct dbuf new_data;


@@ -1396,6 +1407,64 @@ static void *cmd_data (size_t *size)
        return buffer;
 }
 
        return buffer;
 }
 

+static int validate_raw_date(const char *src, char *result, int maxlen)
+{
+       const char *orig_src = src;
+       char *endp, sign;
+
+       strtoul(src, &endp, 10);
+       if (endp == src || *endp != ' ')
+               return -1;
+
+       src = endp + 1;
+       if (*src != '-' && *src != '+')
+               return -1;
+       sign = *src;
+
+       strtoul(src + 1, &endp, 10);
+       if (endp == src || *endp || (endp - orig_src) >= maxlen)
+               return -1;
+
+       strcpy(result, orig_src);
+       return 0;
+}
+
+static char *parse_ident(const char *buf)
+{
+       const char *gt;
+       size_t name_len;
+       char *ident;
+
+       gt = strrchr(buf, '>');
+       if (!gt)
+               die("Missing > in ident string: %s", buf);
+       gt++;
+       if (*gt != ' ')
+               die("Missing space after > in ident string: %s", buf);
+       gt++;
+       name_len = gt - buf;
+       ident = xmalloc(name_len + 24);
+       strncpy(ident, buf, name_len);
+
+       switch (whenspec) {
+       case WHENSPEC_RAW:
+               if (validate_raw_date(gt, ident + name_len, 24) < 0)
+                       die("Invalid raw date \"%s\" in ident: %s", gt, buf);
+               break;
+       case WHENSPEC_RFC2822:
+               if (parse_date(gt, ident + name_len, 24) < 0)
+                       die("Invalid rfc2822 date \"%s\" in ident: %s", gt, buf);
+               break;
+       case WHENSPEC_NOW:
+               if (strcmp("now", gt))
+                       die("Date in ident must be 'now': %s", buf);
+               datestamp(ident + name_len, 24);
+               break;
+       }
+
+       return ident;
+}
+

 static void cmd_new_blob(void)
 {
        size_t l;
 static void cmd_new_blob(void)
 {
        size_t l;


@@ -1655,11 +1724,11 @@ static void cmd_new_commit(void)
        read_next_command();
        cmd_mark();
        if (!strncmp("author ", command_buf.buf, 7)) {
        read_next_command();
        cmd_mark();
        if (!strncmp("author ", command_buf.buf, 7)) {

-               author = strdup(command_buf.buf);
+               author = parse_ident(command_buf.buf + 7);

                read_next_command();
        }
        if (!strncmp("committer ", command_buf.buf, 10)) {
                read_next_command();
        }
        if (!strncmp("committer ", command_buf.buf, 10)) {

-               committer = strdup(command_buf.buf);
+               committer = parse_ident(command_buf.buf + 10);

                read_next_command();
        }
        if (!committer)
                read_next_command();
        }
        if (!committer)


@@ -1692,7 +1761,7 @@ static void cmd_new_commit(void)
        store_tree(&b->branch_tree);
        hashcpy(b->branch_tree.versions[0].sha1,
                b->branch_tree.versions[1].sha1);
        store_tree(&b->branch_tree);
        hashcpy(b->branch_tree.versions[0].sha1,
                b->branch_tree.versions[1].sha1);

-       size_dbuf(&new_data, 97 + msglen
+       size_dbuf(&new_data, 114 + msglen

                + merge_count * 49
                + (author
                        ? strlen(author) + strlen(committer)
                + merge_count * 49
                + (author
                        ? strlen(author) + strlen(committer)


@@ -1708,11 +1777,9 @@ static void cmd_new_commit(void)
                free(merge_list);
                merge_list = next;
        }
                free(merge_list);
                merge_list = next;
        }

-       if (author)
-               sp += sprintf(sp, "%s\n", author);
-       else
-               sp += sprintf(sp, "author %s\n", committer + 10);
-       sp += sprintf(sp, "%s\n\n", committer);
+       sp += sprintf(sp, "author %s\n", author ? author : committer);
+       sp += sprintf(sp, "committer %s\n", committer);
+       *sp++ = '\n';

        memcpy(sp, msg, msglen);
        sp += msglen;
        free(author);
        memcpy(sp, msg, msglen);
        sp += msglen;
        free(author);


@@ -1780,7 +1847,7 @@ static void cmd_new_tag(void)
        /* tagger ... */
        if (strncmp("tagger ", command_buf.buf, 7))
                die("Expected tagger command, got %s", command_buf.buf);
        /* tagger ... */
        if (strncmp("tagger ", command_buf.buf, 7))
                die("Expected tagger command, got %s", command_buf.buf);

-       tagger = strdup(command_buf.buf);
+       tagger = parse_ident(command_buf.buf + 7);

 
        /* tag payload/message */
        read_next_command();
 
        /* tag payload/message */
        read_next_command();


@@ -1792,7 +1859,8 @@ static void cmd_new_tag(void)
        sp += sprintf(sp, "object %s\n", sha1_to_hex(sha1));
        sp += sprintf(sp, "type %s\n", type_names[OBJ_COMMIT]);
        sp += sprintf(sp, "tag %s\n", t->name);
        sp += sprintf(sp, "object %s\n", sha1_to_hex(sha1));
        sp += sprintf(sp, "type %s\n", type_names[OBJ_COMMIT]);
        sp += sprintf(sp, "tag %s\n", t->name);

-       sp += sprintf(sp, "%s\n\n", tagger);
+       sp += sprintf(sp, "tagger %s\n", tagger);
+       *sp++ = '\n';

        memcpy(sp, msg, msglen);
        sp += msglen;
        free(tagger);
        memcpy(sp, msg, msglen);
        sp += msglen;
        free(tagger);


@@ -1835,7 +1903,7 @@ static void cmd_checkpoint(void)
 }
 
 static const char fast_import_usage[] =
 }
 
 static const char fast_import_usage[] =

-"git-fast-import [--depth=n] [--active-branches=n] [--export-marks=marks.file] [--branch-log=log]";
+"git-fast-import [--date-format=f] [--max-pack-size=n] [--depth=n] [--active-branches=n] [--export-marks=marks.file]";

 
 int main(int argc, const char **argv)
 {
 
 int main(int argc, const char **argv)
 {


@@ -1849,6 +1917,17 @@ int main(int argc, const char **argv)
 
                if (*a != '-' || !strcmp(a, "--"))
                        break;
 
                if (*a != '-' || !strcmp(a, "--"))
                        break;

+               else if (!strncmp(a, "--date-format=", 14)) {
+                       const char *fmt = a + 14;
+                       if (!strcmp(fmt, "raw"))
+                               whenspec = WHENSPEC_RAW;
+                       else if (!strcmp(fmt, "rfc2822"))
+                               whenspec = WHENSPEC_RFC2822;
+                       else if (!strcmp(fmt, "now"))
+                               whenspec = WHENSPEC_NOW;
+                       else
+                               die("unknown --date-format argument %s", fmt);
+               }

                else if (!strncmp(a, "--max-pack-size=", 16))
                        max_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;
                else if (!strncmp(a, "--depth=", 8))
                else if (!strncmp(a, "--max-pack-size=", 16))
                        max_packsize = strtoumax(a + 16, NULL, 0) * 1024 * 1024;
                else if (!strncmp(a, "--depth=", 8))

diff --git a/t/t9300-fast-import.sh b/t/t9300-fast-import.sh
index a5cc846b34d521f659e81e4448cdc52d2d3b564d..84b3c12a50c5fb3ea2e98ff5b4a7a441b203dbb6 100755 (executable)
--- a/t/t9300-fast-import.sh
+++ b/t/t9300-fast-import.sh
@@ -240,4 +240,40 @@ test_expect_success \
        'git-cat-file blob branch:newdir/exec.sh >actual &&
         diff -u expect actual'
 
        'git-cat-file blob branch:newdir/exec.sh >actual &&
         diff -u expect actual'
 

+###
+### series E
+###
+
+cat >input <<INPUT_END
+commit refs/heads/branch
+author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> Tue Feb 6 11:22:18 2007 -0500
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> Tue Feb 6 12:35:02 2007 -0500
+data <<COMMIT
+RFC 2822 type date
+COMMIT
+
+from refs/heads/branch^0
+
+INPUT_END
+test_expect_failure \
+    'E: rfc2822 date, --date-format=raw' \
+    'git-fast-import --date-format=raw <input'
+test_expect_success \
+    'E: rfc2822 date, --date-format=rfc2822' \
+    'git-fast-import --date-format=rfc2822 <input'
+test_expect_success \
+       'E: verify pack' \
+       'for p in .git/objects/pack/*.pack;do git-verify-pack $p||exit;done'
+
+cat >expect <<EOF
+author $GIT_AUTHOR_NAME <$GIT_AUTHOR_EMAIL> 1170778938 -0500
+committer $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1170783302 -0500
+
+RFC 2822 type date
+EOF
+test_expect_success \
+       'E: verify commit' \
+       'git-cat-file commit branch | sed 1,2d >actual &&
+       diff -u expect actual'
+

 test_done
 test_done