From e42797f5b6d5ed3b8c894d89493e285c40d58dc8 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Mon, 23 Oct 2006 14:50:18 -0400 Subject: enable index-pack streaming capability A new flag, --stdin, allows for a pack to be received over a stream. When this flag is provided, the pack content is written to either the named pack file or directly to the object repository under the same name as produced by git-repack. The pack index is written as well with the corresponding base name, unless the index name is overriden with -o. With this patch, git-index-pack could be used instead of git-unpack-objects when fetching remote objects but only with non "thin" packs for now. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index 71ce55727..db7af5859 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -8,7 +8,7 @@ git-index-pack - Build pack index file for an existing packed archive SYNOPSIS -------- -'git-index-pack' [-o ] +'git-index-pack' [-o ] { | --stdin [] } DESCRIPTION @@ -29,6 +29,12 @@ OPTIONS fails if the name of packed archive does not end with .pack). +--stdin:: + When this flag is provided, the pack is read from stdin + instead and a copy is then written to . If + is not specified, the pack is written to + objects/pack/ directory of the current git repository with + a default name determined from the pack content. Author ------ -- cgit v1.2.1 From 636171cb80255682bdfc9bf5a98c9e66d4c0444a Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Wed, 25 Oct 2006 23:28:17 -0400 Subject: make index-pack able to complete thin packs. A new flag, --fix-thin, instructs git-index-pack to append any missing objects to a thin pack to make it self contained and indexable. Of course objects missing from the pack must be present elsewhere in the local repository. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index db7af5859..c58287d68 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -8,7 +8,8 @@ git-index-pack - Build pack index file for an existing packed archive SYNOPSIS -------- -'git-index-pack' [-o ] { | --stdin [] } +'git-index-pack' [-o ] +'git-index-pack' --stdin [--fix-thin] [-o ] [] DESCRIPTION @@ -36,6 +37,17 @@ OPTIONS objects/pack/ directory of the current git repository with a default name determined from the pack content. +--fix-thin:: + It is possible for gitlink:git-pack-objects[1] to build + "thin" pack, which records objects in deltified form based on + objects not included in the pack to reduce network traffic. + Those objects are expected to be present on the receiving end + and they must be included in the pack for that pack to be self + contained and indexable. Without this option any attempt to + index a thin pack will fail. This option only makes sense in + conjonction with --stdin. + + Author ------ Written by Sergey Vlasov -- cgit v1.2.1 From 3c9af366469524920864bbc1af4dcfd72029c314 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Wed, 25 Oct 2006 23:32:59 -0400 Subject: add progress status to index-pack This is more interesting to look at when performing a big fetch. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) (limited to 'Documentation') diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index c58287d68..9fa4847d5 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -8,8 +8,8 @@ git-index-pack - Build pack index file for an existing packed archive SYNOPSIS -------- -'git-index-pack' [-o ] -'git-index-pack' --stdin [--fix-thin] [-o ] [] +'git-index-pack' [-v] [-o ] +'git-index-pack' --stdin [--fix-thin] [-v] [-o ] [] DESCRIPTION @@ -22,6 +22,9 @@ objects/pack/ directory of a git repository. OPTIONS ------- +-v:: + Be verbose about what is going on, including progress status. + -o :: Write the generated pack index into the specified file. Without this option the name of pack index -- cgit v1.2.1 From b8077709243138c3d8cc1c096c06a95b250a9001 Mon Sep 17 00:00:00 2001 From: Shawn Pearce Date: Sun, 29 Oct 2006 04:41:59 -0500 Subject: Teach git-index-pack how to keep a pack file. To prevent a race condition between `index-pack --stdin` and `repack -a -d` where the repack deletes the newly created pack file before any refs are updated to reference objects contained within it we mark the pack file as one that should be kept. This removes it from the list of packs that `repack -a -d` will consider for removal. Callers such as `receive-pack` which want to invoke `index-pack` should use this new --keep option to prevent the newly created pack and index file pair from being deleted before they have finished any related ref updates. Only after all ref updates have been finished should the associated .keep file be removed. Signed-off-by: Shawn O. Pearce Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) (limited to 'Documentation') diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index 9fa4847d5..1235416e0 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -9,7 +9,7 @@ git-index-pack - Build pack index file for an existing packed archive SYNOPSIS -------- 'git-index-pack' [-v] [-o ] -'git-index-pack' --stdin [--fix-thin] [-v] [-o ] [] +'git-index-pack' --stdin [--fix-thin] [--keep] [-v] [-o ] [] DESCRIPTION @@ -38,7 +38,10 @@ OPTIONS instead and a copy is then written to . If is not specified, the pack is written to objects/pack/ directory of the current git repository with - a default name determined from the pack content. + a default name determined from the pack content. If + is not specified consider using --keep to + prevent a race condition between this process and + gitlink::git-repack[1] . --fix-thin:: It is possible for gitlink:git-pack-objects[1] to build @@ -48,7 +51,22 @@ OPTIONS and they must be included in the pack for that pack to be self contained and indexable. Without this option any attempt to index a thin pack will fail. This option only makes sense in - conjonction with --stdin. + conjunction with --stdin. + +--keep:: + Before moving the index into its final destination + create an empty .keep file for the associated pack file. + This option is usually necessary with --stdin to prevent a + simultaneous gitlink:git-repack[1] process from deleting + the newly constructed pack and index before refs can be + updated to use objects contained in the pack. + +--keep='why':: + Like --keep create a .keep file before moving the index into + its final destination, but rather than creating an empty file + place 'why' followed by an LF into the .keep file. The 'why' + message can later be searched for within all .keep files to + locate any which have outlived their usefulness. Author -- cgit v1.2.1 From fc04c412d8d2412e97bb2a664a1746e333dfd9ae Mon Sep 17 00:00:00 2001 From: Shawn Pearce Date: Wed, 1 Nov 2006 17:06:21 -0500 Subject: Teach receive-pack how to keep pack files based on object count. Since keeping a pushed pack or exploding it into loose objects should be a local repository decision this teaches receive-pack to decide if it should call unpack-objects or index-pack --stdin --fix-thin based on the setting of receive.unpackLimit and the number of objects contained in the received pack. If the number of objects (hdr_entries) in the received pack is below the value of receive.unpackLimit (which is 5000 by default) then we unpack-objects as we have in the past. If the hdr_entries >= receive.unpackLimit then we call index-pack and ask it to include our pid and hostname in the .keep file to make it easier to identify why a given pack has been kept in the repository. Currently this leaves every received pack as a kept pack. We really don't want that as received packs will tend to be small. Instead we want to delete the .keep file automatically after all refs have been updated. That is being left as room for future improvement. Signed-off-by: Shawn O. Pearce Signed-off-by: Junio C Hamano --- Documentation/config.txt | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/config.txt b/Documentation/config.txt index d9e73da2a..9d3c71c3b 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -301,7 +301,16 @@ imap:: The configuration variables in the 'imap' section are described in gitlink:git-imap-send[1]. -receive.denyNonFastforwads:: +receive.unpackLimit:: + If the number of objects received in a push is below this + limit then the objects will be unpacked into loose object + files. However if the number of received objects equals or + exceeds this limit then the received pack will be stored as + a pack, after adding any missing delta bases. Storing the + pack from a push can make the push operation complete faster, + especially on slow filesystems. + +receive.denyNonFastForwards:: If set to true, git-receive-pack will deny a ref update which is not a fast forward. Use this to prevent such an update via a push, even if that push is forced. This configuration variable is -- cgit v1.2.1 From da093d37506f034bfb14e6de26cce9c14b1a1216 Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Wed, 1 Nov 2006 17:06:23 -0500 Subject: improve fetch-pack's handling of kept packs Since functions in fetch-clone.c were only used from fetch-pack.c, its content has been merged with fetch-pack.c. This allows for better coupling of features with much simpler implementations. One new thing is that the (abscence of) --thin also enforce it on index-pack now, such that index-pack will abort if a thin pack was _not_ asked for. The -k or --keep, when provided twice, now causes the fetched pack to be left as a kept pack just like receive-pack currently does. Eventually this will be used to close a race against concurrent repacking. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- Documentation/git-fetch-pack.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) (limited to 'Documentation') diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt index bff9aa693..3e6cd880b 100644 --- a/Documentation/git-fetch-pack.txt +++ b/Documentation/git-fetch-pack.txt @@ -32,7 +32,8 @@ OPTIONS -k:: Do not invoke 'git-unpack-objects' on received data, but create a single packfile out of it instead, and store it - in the object database. + in the object database. If provided twice then the pack is + locked against repacking. --exec=:: Use this to specify the path to 'git-upload-pack' on the -- cgit v1.2.1 From 576162a45f35e157427300066b0ff566ff698a0f Mon Sep 17 00:00:00 2001 From: Nicolas Pitre Date: Wed, 1 Nov 2006 17:06:25 -0500 Subject: remove .keep pack lock files when done with refs update This makes both git-fetch and git-push (fetch-pack and receive-pack) safe against a possible race with aparallel git-repack -a -d that could prune the new pack while it is not yet referenced, and remove the .keep file after refs have been updated. Signed-off-by: Nicolas Pitre Signed-off-by: Junio C Hamano --- Documentation/git-index-pack.txt | 11 +++++++++++ 1 file changed, 11 insertions(+) (limited to 'Documentation') diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index 1235416e0..2229ee86b 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -69,6 +69,17 @@ OPTIONS locate any which have outlived their usefulness. +Note +---- + +Once the index has been created, the list of object names is sorted +and the SHA1 hash of that list is printed to stdout. If --stdin was +also used then this is prefixed by either "pack\t", or "keep\t" if a +new .keep file was successfully created. This is useful to remove a +.keep file used as a lock to prevent the race with gitlink:git-repack[1] +mentioned above. + + Author ------ Written by Sergey Vlasov -- cgit v1.2.1