From 78d1e84fe5586ed45740b915b52e6856bb3f1c44 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Mon, 9 Apr 2007 01:06:31 -0400 Subject: compute a CRC32 for each object as stored in a pack The most important optimization for performance when repacking is the ability to reuse data from a previous pack as is and bypass any delta or even SHA1 computation by simply copying the raw data from one pack to another directly. The problem with this is that any data corruption within a copied object would go unnoticed and the new (repacked) pack would be self-consistent with its own checksum despite containing a corrupted object. This is a real issue that already happened at least once in the past. In some attempt to prevent this, we validate the copied data by inflating it and making sure no error is signaled by zlib. But this is still not perfect as a significant portion of a pack content is made of object headers and references to delta base objects which are not deflated and therefore not validated when repacking actually making the pack data reuse still not as safe as it could be. Of course a full SHA1 validation could be performed, but that implies full data inflating and delta replaying which is extremely costly, which cost the data reuse optimization was designed to avoid in the first place. So the best solution to this is simply to store a CRC32 of the raw pack data for each object in the pack index. This way any object in a pack can be validated before being copied as is in another pack, including header and any other non deflated data. Why CRC32 instead of a faster checksum like Adler32? Quoting Wikipedia: Jonathan Stone discovered in 2001 that Adler-32 has a weakness for very short messages. He wrote "Briefly, the problem is that, for very short packets, Adler32 is guaranteed to give poor coverage of the available bits. Don't take my word for it, ask Mark Adler. :-)" The problem is that sum A does not wrap for short messages. The maximum value of A for a 128-byte message is 32640, which is below the value 65521 used by the modulo operation. An extended explanation can be found in RFC 3309, which mandates the use of CRC32 instead of Adler-32 for SCTP, the Stream Control Transmission Protocol. In the context of a GIT pack, we have lots of small objects, especially deltas, which are likely to be quite small and in a size range for which Adler32 is dimed not to be sufficient. Another advantage of CRC32 is the possibility for recovery from certain types of small corruptions like single bit errors which are the most probable type of corruptions. OK what this patch does is to compute the CRC32 of each object written to a pack within pack-objects. It is not written to the index yet and it is obviously not validated when reusing pack data yet either. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index b7174c6c0..7c806ada4 100644 --- a/csum-file.c +++ b/csum-file.c @@ -49,6 +49,8 @@ int sha1close(struct sha1file *f, unsigned char *result, int update) int sha1write(struct sha1file *f, void *buf, unsigned int count) { + if (f->do_crc) + f->crc32 = crc32(f->crc32, buf, count); while (count) { unsigned offset = f->offset; unsigned left = sizeof(f->buffer) - offset; @@ -91,6 +93,7 @@ struct sha1file *sha1create(const char *fmt, ...) f->fd = fd; f->error = 0; f->offset = 0; + f->do_crc = 0; SHA1_Init(&f->ctx); return f; } @@ -111,6 +114,7 @@ struct sha1file *sha1fd(int fd, const char *name) f->fd = fd; f->error = 0; f->offset = 0; + f->do_crc = 0; SHA1_Init(&f->ctx); return f; } @@ -143,4 +147,14 @@ int sha1write_compressed(struct sha1file *f, void *in, unsigned int size) return size; } +void crc32_begin(struct sha1file *f) +{ + f->crc32 = crc32(0, Z_NULL, 0); + f->do_crc = 1; +} +uint32_t crc32_end(struct sha1file *f) +{ + f->do_crc = 0; + return f->crc32; +} -- cgit v1.2.1 From 960ccca6803c9fb57429d43572a9545a96107e32 Mon Sep 17 00:00:00 2001 From: Dana How Date: Wed, 9 May 2007 13:56:50 -0700 Subject: Custom compression levels for objects and packs Add config variables pack.compression and core.loosecompression , and switch --compression=level to pack-objects. Loose objects will be compressed using core.loosecompression if set, else core.compression if set, else Z_BEST_SPEED. Packed objects will be compressed using --compression=level if seen, else pack.compression if set, else core.compression if set, else Z_DEFAULT_COMPRESSION. This is the "pack compression level". Loose objects added to a pack undeltified will be recompressed to the pack compression level if it is unequal to the current loose compression level by the preceding rules, or if the loose object was written while core.legacyheaders = true. Newly deltified loose objects are always compressed to the current pack compression level. Previously packed objects added to a pack are recompressed to the current pack compression level exactly when their deltification status changes, since the previous pack data cannot be reused. In either case, the --no-reuse-object switch from the first patch below will always force recompression to the current pack compression level, instead of assuming the pack compression level hasn't changed and pack data can be reused when possible. This applies on top of the following patches from Nicolas Pitre: [PATCH] allow for undeltified objects not to be reused [PATCH] make "repack -f" imply "pack-objects --no-reuse-object" Signed-off-by: Dana L. How Signed-off-by: Junio C Hamano --- csum-file.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 7c806ada4..7088f6e93 100644 --- a/csum-file.c +++ b/csum-file.c @@ -119,14 +119,14 @@ struct sha1file *sha1fd(int fd, const char *name) return f; } -int sha1write_compressed(struct sha1file *f, void *in, unsigned int size) +int sha1write_compressed(struct sha1file *f, void *in, unsigned int size, int level) { z_stream stream; unsigned long maxsize; void *out; memset(&stream, 0, sizeof(stream)); - deflateInit(&stream, zlib_compression_level); + deflateInit(&stream, level); maxsize = deflateBound(&stream, size); out = xmalloc(maxsize); -- cgit v1.2.1 From f02153696fc319c3c6dba2c27a5f7aed98eec465 Mon Sep 17 00:00:00 2001 From: "Dana L. How" Date: Sun, 13 May 2007 11:28:19 -0700 Subject: Alter sha1close() 3rd argument to request flush only update=0 suppressed writing the final SHA-1 but was not used. Now final=0 suppresses SHA-1 finalization, SHA-1 writing, and closing -- in other words, only flush the buffer. Signed-off-by: Dana L. How Signed-off-by: Junio C Hamano --- csum-file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 7088f6e93..510934262 100644 --- a/csum-file.c +++ b/csum-file.c @@ -29,18 +29,20 @@ static void sha1flush(struct sha1file *f, unsigned int count) } } -int sha1close(struct sha1file *f, unsigned char *result, int update) +int sha1close(struct sha1file *f, unsigned char *result, int final) { unsigned offset = f->offset; if (offset) { SHA1_Update(&f->ctx, f->buffer, offset); sha1flush(f, offset); + f->offset = 0; } + if (!final) + return 0; /* only want to flush (no checksum write, no close) */ SHA1_Final(f->buffer, &f->ctx); if (result) hashcpy(result, f->buffer); - if (update) - sha1flush(f, 20); + sha1flush(f, 20); if (close(f->fd)) die("%s: sha1 file error on close (%s)", f->name, strerror(errno)); free(f); -- cgit v1.2.1 From 4175e9e3a8734be1e96e385b0fa2428b86ed5809 Mon Sep 17 00:00:00 2001 From: Junio C Hamano Date: Wed, 13 Jun 2007 01:42:05 -0700 Subject: More static There still are quite a few symbols that ought to be static. Signed-off-by: Junio C Hamano --- csum-file.c | 55 ------------------------------------------------------- 1 file changed, 55 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 510934262..9ab997120 100644 --- a/csum-file.c +++ b/csum-file.c @@ -73,33 +73,6 @@ int sha1write(struct sha1file *f, void *buf, unsigned int count) return 0; } -struct sha1file *sha1create(const char *fmt, ...) -{ - struct sha1file *f; - unsigned len; - va_list arg; - int fd; - - f = xmalloc(sizeof(*f)); - - va_start(arg, fmt); - len = vsnprintf(f->name, sizeof(f->name), fmt, arg); - va_end(arg); - if (len >= PATH_MAX) - die("you wascally wabbit, you"); - f->namelen = len; - - fd = open(f->name, O_CREAT | O_EXCL | O_WRONLY, 0666); - if (fd < 0) - die("unable to open %s (%s)", f->name, strerror(errno)); - f->fd = fd; - f->error = 0; - f->offset = 0; - f->do_crc = 0; - SHA1_Init(&f->ctx); - return f; -} - struct sha1file *sha1fd(int fd, const char *name) { struct sha1file *f; @@ -121,34 +94,6 @@ struct sha1file *sha1fd(int fd, const char *name) return f; } -int sha1write_compressed(struct sha1file *f, void *in, unsigned int size, int level) -{ - z_stream stream; - unsigned long maxsize; - void *out; - - memset(&stream, 0, sizeof(stream)); - deflateInit(&stream, level); - maxsize = deflateBound(&stream, size); - out = xmalloc(maxsize); - - /* Compress it */ - stream.next_in = in; - stream.avail_in = size; - - stream.next_out = out; - stream.avail_out = maxsize; - - while (deflate(&stream, Z_FINISH) == Z_OK) - /* nothing */; - deflateEnd(&stream); - - size = stream.total_out; - sha1write(f, out, size); - free(out); - return size; -} - void crc32_begin(struct sha1file *f) { f->crc32 = crc32(0, Z_NULL, 0); -- cgit v1.2.1 From 7ba502c47bda21d060844863991083f4c319d436 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Tue, 16 Oct 2007 21:55:48 -0400 Subject: pack-objects.c: fix some global variable abuse and memory leaks To keep things well layered, sha1close() now returns the file descriptor when it doesn't close the file. An ugly cast was added to the return of write_idx_file() to avoid a warning. A proper fix will come separately. Signed-off-by: Nicolas Pitre Signed-off-by: Shawn O. Pearce --- csum-file.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 9ab997120..9929991de 100644 --- a/csum-file.c +++ b/csum-file.c @@ -31,22 +31,27 @@ static void sha1flush(struct sha1file *f, unsigned int count) int sha1close(struct sha1file *f, unsigned char *result, int final) { + int fd; unsigned offset = f->offset; if (offset) { SHA1_Update(&f->ctx, f->buffer, offset); sha1flush(f, offset); f->offset = 0; } - if (!final) - return 0; /* only want to flush (no checksum write, no close) */ - SHA1_Final(f->buffer, &f->ctx); - if (result) - hashcpy(result, f->buffer); - sha1flush(f, 20); - if (close(f->fd)) - die("%s: sha1 file error on close (%s)", f->name, strerror(errno)); + if (final) { + /* write checksum and close fd */ + SHA1_Final(f->buffer, &f->ctx); + if (result) + hashcpy(result, f->buffer); + sha1flush(f, 20); + if (close(f->fd)) + die("%s: sha1 file error on close (%s)", + f->name, strerror(errno)); + fd = 0; + } else + fd = f->fd; free(f); - return 0; + return fd; } int sha1write(struct sha1file *f, void *buf, unsigned int count) -- cgit v1.2.1 From 2a128d63dc140e8daeaf68659d9263b7389b7d1b Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Tue, 30 Oct 2007 17:06:21 -0400 Subject: add throughput display to git-push This one triggers only when git-pack-objects is called with --all-progress and --stdout which is the combination used by git-push. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 9929991de..3729e73e1 100644 --- a/csum-file.c +++ b/csum-file.c @@ -8,6 +8,7 @@ * able to verify hasn't been messed with afterwards. */ #include "cache.h" +#include "progress.h" #include "csum-file.h" static void sha1flush(struct sha1file *f, unsigned int count) @@ -17,6 +18,7 @@ static void sha1flush(struct sha1file *f, unsigned int count) for (;;) { int ret = xwrite(f->fd, buf, count); if (ret > 0) { + display_throughput(f->tp, ret); buf = (char *) buf + ret; count -= ret; if (count) @@ -79,6 +81,11 @@ int sha1write(struct sha1file *f, void *buf, unsigned int count) } struct sha1file *sha1fd(int fd, const char *name) +{ + return sha1fd_throughput(fd, name, NULL); +} + +struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp) { struct sha1file *f; unsigned len; @@ -94,6 +101,7 @@ struct sha1file *sha1fd(int fd, const char *name) f->fd = fd; f->error = 0; f->offset = 0; + f->tp = tp; f->do_crc = 0; SHA1_Init(&f->ctx); return f; -- cgit v1.2.1 From 218558af599c01e5dec17a7399d9188a76c50203 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Sun, 4 Nov 2007 22:15:41 -0500 Subject: make display of total transferred more accurate The throughput display needs a delay period before accounting and displaying anything. Yet it might be called after some amount of data has already been transferred. The display of total data is therefore accounted late and therefore smaller than the reality. Let's call display_throughput() with an absolute amount of transferred data instead of a relative number, and let the throughput code find the relative amount of data by itself as needed. This way the displayed total is always exact. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 3729e73e1..b445e6a2e 100644 --- a/csum-file.c +++ b/csum-file.c @@ -18,7 +18,8 @@ static void sha1flush(struct sha1file *f, unsigned int count) for (;;) { int ret = xwrite(f->fd, buf, count); if (ret > 0) { - display_throughput(f->tp, ret); + f->total += ret; + display_throughput(f->tp, f->total); buf = (char *) buf + ret; count -= ret; if (count) @@ -101,6 +102,7 @@ struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp f->fd = fd; f->error = 0; f->offset = 0; + f->total = 0; f->tp = tp; f->do_crc = 0; SHA1_Init(&f->ctx); -- cgit v1.2.1 From ec640ed1cf1d62730555705ec18b785c43e81f62 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Sun, 4 Nov 2007 22:54:50 -0500 Subject: remove dead code from the csum-file interface The provided name argument is always constant and valid in every caller's context, so no need to have an array of PATH_MAX chars to copy it into when a simple pointer will do. Unfortunately that means getting rid of wascally wabbits too. The 'error' field is also unused. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index b445e6a2e..9728a9954 100644 --- a/csum-file.c +++ b/csum-file.c @@ -88,22 +88,12 @@ struct sha1file *sha1fd(int fd, const char *name) struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp) { - struct sha1file *f; - unsigned len; - - f = xmalloc(sizeof(*f)); - - len = strlen(name); - if (len >= PATH_MAX) - die("you wascally wabbit, you"); - f->namelen = len; - memcpy(f->name, name, len+1); - + struct sha1file *f = xmalloc(sizeof(*f)); f->fd = fd; - f->error = 0; f->offset = 0; f->total = 0; f->tp = tp; + f->name = name; f->do_crc = 0; SHA1_Init(&f->ctx); return f; -- cgit v1.2.1 From 4c81b03e30d13dbc93ea7071438ef2da0acd4189 Mon Sep 17 00:00:00 2001 From: Linus Torvalds Date: Fri, 30 May 2008 08:42:16 -0700 Subject: Make pack creation always fsync() the result This means that we can depend on packs always being stable on disk, simplifying a lot of the object serialization worries. And unlike loose objects, serializing pack creation IO isn't going to be a performance killer. Signed-off-by: Linus Torvalds Signed-off-by: Junio C Hamano --- csum-file.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 9728a9954..ace64f165 100644 --- a/csum-file.c +++ b/csum-file.c @@ -32,21 +32,24 @@ static void sha1flush(struct sha1file *f, unsigned int count) } } -int sha1close(struct sha1file *f, unsigned char *result, int final) +int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) { int fd; unsigned offset = f->offset; + if (offset) { SHA1_Update(&f->ctx, f->buffer, offset); sha1flush(f, offset); f->offset = 0; } - if (final) { + if (flags & (CSUM_CLOSE | CSUM_FSYNC)) { /* write checksum and close fd */ SHA1_Final(f->buffer, &f->ctx); if (result) hashcpy(result, f->buffer); sha1flush(f, 20); + if (flags & CSUM_FSYNC) + fsync_or_die(f->fd, f->name); if (close(f->fd)) die("%s: sha1 file error on close (%s)", f->name, strerror(errno)); -- cgit v1.2.1 From ac0463ed44e859c84e354cd0ae47d9b5b124e112 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Fri, 29 Aug 2008 16:08:00 -0400 Subject: pack-objects: use fixup_pack_header_footer()'s validation mode When limiting the pack size, a new header has to be written to the pack and a new SHA1 computed. Make sure that the SHA1 of what is being read back matches the SHA1 of what was written. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index ace64f165..28389541a 100644 --- a/csum-file.c +++ b/csum-file.c @@ -42,11 +42,11 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) sha1flush(f, offset); f->offset = 0; } + SHA1_Final(f->buffer, &f->ctx); + if (result) + hashcpy(result, f->buffer); if (flags & (CSUM_CLOSE | CSUM_FSYNC)) { /* write checksum and close fd */ - SHA1_Final(f->buffer, &f->ctx); - if (result) - hashcpy(result, f->buffer); sha1flush(f, 20); if (flags & CSUM_FSYNC) fsync_or_die(f->fd, f->name); -- cgit v1.2.1 From a8032d12241f2226a6ab98272b12a1d98ca446a5 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Tue, 2 Sep 2008 10:22:20 -0400 Subject: sha1write: don't copy full sized buffers No need to memcpy() source buffer data when we might just process the data in place instead of accumulating it into a separate buffer. This is the case when a whole buffer would have been copied, summed, written out and then discarded right away. Also move the CRC32 processing within the loop so the data is more likely to remain in the L1 CPU cache between the CRC32 sum, SHA1 sum and the write call. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- csum-file.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 28389541a..bb70c75ee 100644 --- a/csum-file.c +++ b/csum-file.c @@ -11,10 +11,8 @@ #include "progress.h" #include "csum-file.h" -static void sha1flush(struct sha1file *f, unsigned int count) +static void sha1flush(struct sha1file *f, void *buf, unsigned int count) { - void *buf = f->buffer; - for (;;) { int ret = xwrite(f->fd, buf, count); if (ret > 0) { @@ -39,7 +37,7 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) if (offset) { SHA1_Update(&f->ctx, f->buffer, offset); - sha1flush(f, offset); + sha1flush(f, f->buffer, offset); f->offset = 0; } SHA1_Final(f->buffer, &f->ctx); @@ -47,7 +45,7 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) hashcpy(result, f->buffer); if (flags & (CSUM_CLOSE | CSUM_FSYNC)) { /* write checksum and close fd */ - sha1flush(f, 20); + sha1flush(f, f->buffer, 20); if (flags & CSUM_FSYNC) fsync_or_die(f->fd, f->name); if (close(f->fd)) @@ -62,21 +60,30 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) int sha1write(struct sha1file *f, void *buf, unsigned int count) { - if (f->do_crc) - f->crc32 = crc32(f->crc32, buf, count); while (count) { unsigned offset = f->offset; unsigned left = sizeof(f->buffer) - offset; unsigned nr = count > left ? left : count; + void *data; + + if (f->do_crc) + f->crc32 = crc32(f->crc32, buf, nr); + + if (nr == sizeof(f->buffer)) { + /* process full buffer directly without copy */ + data = buf; + } else { + memcpy(f->buffer + offset, buf, nr); + data = f->buffer; + } - memcpy(f->buffer + offset, buf, nr); count -= nr; offset += nr; buf = (char *) buf + nr; left -= nr; if (!left) { - SHA1_Update(&f->ctx, f->buffer, offset); - sha1flush(f, offset); + SHA1_Update(&f->ctx, data, offset); + sha1flush(f, data, offset); offset = 0; } f->offset = offset; -- cgit v1.2.1 From 9126f0091f271f090cc030a788219574ab0fea97 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Wed, 1 Oct 2008 14:05:20 -0400 Subject: fix openssl headers conflicting with custom SHA1 implementations On ARM I have the following compilation errors: CC fast-import.o In file included from cache.h:8, from builtin.h:6, from fast-import.c:142: arm/sha1.h:14: error: conflicting types for 'SHA_CTX' /usr/include/openssl/sha.h:105: error: previous declaration of 'SHA_CTX' was here arm/sha1.h:16: error: conflicting types for 'SHA1_Init' /usr/include/openssl/sha.h:115: error: previous declaration of 'SHA1_Init' was here arm/sha1.h:17: error: conflicting types for 'SHA1_Update' /usr/include/openssl/sha.h:116: error: previous declaration of 'SHA1_Update' was here arm/sha1.h:18: error: conflicting types for 'SHA1_Final' /usr/include/openssl/sha.h:117: error: previous declaration of 'SHA1_Final' was here make: *** [fast-import.o] Error 1 This is because openssl header files are always included in git-compat-util.h since commit 684ec6c63c whenever NO_OPENSSL is not set, which somehow brings in clashing with the custom ARM version. Compilation of git is probably broken on PPC too for the same reason. Turns out that the only file requiring openssl/ssl.h and openssl/err.h is imap-send.c. But only moving those problematic includes there doesn't solve the issue as it also includes cache.h which brings in the conflicting local SHA1 header file. As suggested by Jeff King, the best solution is to rename our references to SHA1 functions and structure to something git specific, and define those according to the implementation used. Signed-off-by: Nicolas Pitre Signed-off-by: Shawn O. Pearce --- csum-file.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index bb70c75ee..717d29fc0 100644 --- a/csum-file.c +++ b/csum-file.c @@ -36,11 +36,11 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) unsigned offset = f->offset; if (offset) { - SHA1_Update(&f->ctx, f->buffer, offset); + git_SHA1_Update(&f->ctx, f->buffer, offset); sha1flush(f, f->buffer, offset); f->offset = 0; } - SHA1_Final(f->buffer, &f->ctx); + git_SHA1_Final(f->buffer, &f->ctx); if (result) hashcpy(result, f->buffer); if (flags & (CSUM_CLOSE | CSUM_FSYNC)) { @@ -82,7 +82,7 @@ int sha1write(struct sha1file *f, void *buf, unsigned int count) buf = (char *) buf + nr; left -= nr; if (!left) { - SHA1_Update(&f->ctx, data, offset); + git_SHA1_Update(&f->ctx, data, offset); sha1flush(f, data, offset); offset = 0; } @@ -105,7 +105,7 @@ struct sha1file *sha1fd_throughput(int fd, const char *name, struct progress *tp f->tp = tp; f->name = name; f->do_crc = 0; - SHA1_Init(&f->ctx); + git_SHA1_Init(&f->ctx); return f; } -- cgit v1.2.1 From 838cd34664422863096f1a089f779bed1f00edf6 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Thu, 9 Oct 2008 22:08:51 -0400 Subject: fix pread()'s short read in index-pack Since v1.6.0.2~13^2~ the completion of a thin pack uses sha1write() for its ability to compute a SHA1 on the written data. This also provides data buffering which, along with commit 92392b4a45, will confuse pread() whenever an appended object is 1) freed due to memory pressure because of the depth-first delta processing, and 2) needed again because it has many delta children, and 3) its data is still buffered by sha1write(). Let's fix the issue by simply forcing cached data out when such an object is written so it can be pread()'d at leisure. Signed-off-by: Nicolas Pitre Signed-off-by: Shawn O. Pearce --- csum-file.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 28389541a..cfc1ac42b 100644 --- a/csum-file.c +++ b/csum-file.c @@ -11,7 +11,7 @@ #include "progress.h" #include "csum-file.h" -static void sha1flush(struct sha1file *f, unsigned int count) +static void flush(struct sha1file *f, unsigned int count) { void *buf = f->buffer; @@ -32,22 +32,28 @@ static void sha1flush(struct sha1file *f, unsigned int count) } } -int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) +void sha1flush(struct sha1file *f) { - int fd; unsigned offset = f->offset; if (offset) { SHA1_Update(&f->ctx, f->buffer, offset); - sha1flush(f, offset); + flush(f, offset); f->offset = 0; } +} + +int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) +{ + int fd; + + sha1flush(f); SHA1_Final(f->buffer, &f->ctx); if (result) hashcpy(result, f->buffer); if (flags & (CSUM_CLOSE | CSUM_FSYNC)) { /* write checksum and close fd */ - sha1flush(f, 20); + flush(f, 20); if (flags & CSUM_FSYNC) fsync_or_die(f->fd, f->name); if (close(f->fd)) @@ -76,7 +82,7 @@ int sha1write(struct sha1file *f, void *buf, unsigned int count) left -= nr; if (!left) { SHA1_Update(&f->ctx, f->buffer, offset); - sha1flush(f, offset); + flush(f, offset); offset = 0; } f->offset = offset; -- cgit v1.2.1 From d824cbba02a4061400a0e382f9bd241fbbff34f0 Mon Sep 17 00:00:00 2001 From: Thomas Rast Date: Sat, 27 Jun 2009 17:58:46 +0200 Subject: Convert existing die(..., strerror(errno)) to die_errno() Change calls to die(..., strerror(errno)) to use the new die_errno(). In the process, also make slight style adjustments: at least state _something_ about the function that failed (instead of just printing the pathname), and put paths in single quotes. Signed-off-by: Thomas Rast Signed-off-by: Junio C Hamano --- csum-file.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) (limited to 'csum-file.c') diff --git a/csum-file.c b/csum-file.c index 2ddb12a0b..4d50cc5ce 100644 --- a/csum-file.c +++ b/csum-file.c @@ -26,7 +26,7 @@ static void flush(struct sha1file *f, void * buf, unsigned int count) } if (!ret) die("sha1 file '%s' write error. Out of diskspace", f->name); - die("sha1 file '%s' write error (%s)", f->name, strerror(errno)); + die_errno("sha1 file '%s' write error", f->name); } } @@ -55,8 +55,7 @@ int sha1close(struct sha1file *f, unsigned char *result, unsigned int flags) if (flags & CSUM_FSYNC) fsync_or_die(f->fd, f->name); if (close(f->fd)) - die("%s: sha1 file error on close (%s)", - f->name, strerror(errno)); + die_errno("%s: sha1 file error on close", f->name); fd = 0; } else fd = f->fd; -- cgit v1.2.1