aboutsummaryrefslogtreecommitdiff
path: root/sha1_file.c
Commit message (Collapse)AuthorAge
* Avoid errors and warnings when attempting to do I/O on zero bytesEric Wong2007-01-11
| | | | | | | | | | | Unfortunately, while {read,write}_in_full do take into account zero-sized reads/writes; their die and whine variants do not. I have a repository where there are zero-sized files in the history that was triggering these things. Signed-off-by: Eric Wong <normalperson@yhbt.net> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Fix warnings in sha1_file.c - use C99 printf format if availablePavel Roskin2007-01-09
| | | | | Signed-off-by: Pavel Roskin <proski@gnu.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* short i/o: fix calls to write to use xwrite or write_in_fullAndy Whitcroft2007-01-08
| | | | | | | | | | | | | | We have a number of badly checked write() calls. Often we are expecting write() to write exactly the size we requested or fail, this fails to handle interrupts or short writes. Switch to using the new write_in_full(). Otherwise we at a minimum need to check for EINTR and EAGAIN, where this is appropriate use xwrite(). Note, the changes to config handling are much larger and handled in the next patch in the sequence. Signed-off-by: Andy Whitcroft <apw@shadowen.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* short i/o: fix calls to read to use xread or read_in_fullAndy Whitcroft2007-01-08
| | | | | | | | | | | We have a number of badly checked read() calls. Often we are expecting read() to read exactly the size we requested or fail, this fails to handle interrupts or short reads. Add a read_in_full() providing those semantics. Otherwise we at a minimum need to check for EINTR and EAGAIN, where this is appropriate use xread(). Signed-off-by: Andy Whitcroft <apw@shadowen.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* mmap: set FD_CLOEXEC for file descriptors we keep open for mmap()Junio C Hamano2006-12-29
| | | | | | | | I do not have any proof that this matters to any existing problems I am seeing, but I do not think of any reason not to do this. Signed-off-by: Junio C Hamano <junkio@cox.net>
* Replace mmap with xmmap, better handling MAP_FAILED.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | In some cases we did not even bother to check the return value of mmap() and just assume it worked. This is bad, because if we are out of virtual address space the kernel returned MAP_FAILED and we would attempt to dereference that address, segfaulting without any real error output to the user. We are replacing all calls to mmap() with xmmap() and moving all MAP_FAILED checking into that single location. If a mmap call fails we try to release enough least-recently-used pack windows to possibly succeed, then retry the mmap() attempt. If we cannot mmap even after releasing pack memory then we die() as none of our callers have any reasonable recovery strategy for a failed mmap. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Release pack windows before reporting out of memory.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | If we are about to fail because this process has run out of memory we should first try to automatically control our appetite for address space by releasing enough least-recently-used pack windows to gain back enough memory such that we might actually be able to meet the current allocation request. This should help users who have fairly large repositories but are working on systems with relatively small virtual address space. Many times we see reports on the mailing list of these users running out of memory during various Git operations. Dynamically decreasing the amount of pack memory used when the demand for heap memory is increasing is an intelligent solution to this problem. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Create pack_report() as a debugging aid.Shawn O. Pearce2006-12-29
| | | | | | | | | | Much like the alloc_report() function can be useful to report on object allocation statistics while debugging the new pack_report() function can be useful to report on the behavior of the mmap window code used for packfile access. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Support unmapping windows on 'temporary' packfiles.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a command opens a packfile for only temporary access and does not install the struct packed_git* into the global packed_git list then we are unable to unmap any inactive windows within that packed_git, causing the overall process to exceed core.packedGitLimit. We cannot force the callers to install their temporary packfile into the packed_git chain as doing so would allow that (possibly corrupt but currently being verified) temporary packfile to become part of the local ODB, which may allow it to be considered for object resolution when it may not actually be a valid packfile. So to support unmapping the windows of these temporary packfiles we also scan the windows of the struct packed_git which was supplied to use_pack(). Since commands only work with one temporary packfile at a time scanning the one supplied to use_pack() and all packs installed into packed_git should cover everything available in memory. We also have to be careful to not close the file descriptor of the packed_git which was handed to use_pack() when all of that packfile's windows have been unmapped, as we are already past the open call that would open the packfile and need the file descriptor to be ready for mmap() after unuse_one_window returns. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Improve error message when packfile mmap fails.Shawn O. Pearce2006-12-29
| | | | | | | | | | If we are unable to mmap the a region of the packfile with the mmap() system call there may be a good reason why, such as a closed file descriptor or out of address space. Reporting the system level error message can help to debug such problems. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Fully activate the sliding window pack access.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This finally turns on the sliding window behavior for packfile data access by mapping limited size windows and chaining them under the packed_git->windows list. We consider a given byte offset to be within the window only if there would be at least 20 bytes (one hash worth of data) accessible after the requested offset. This range selection relates to the contract that use_pack() makes with its callers, allowing them to access one hash or one object header without needing to call use_pack() for every byte of data obtained. In the worst case scenario we will map the same page of data twice into memory: once at the end of one window and once again at the start of the next window. This duplicate page mapping will happen only when an object header or a delta base reference is spanned over the end of a window and is always limited to just one page of duplication, as no sane operating system will ever have a page size smaller than a hash. I am assuming that the possible wasted page of virtual address space is going to perform faster than the alternatives, which would be to copy the object header or ref delta into a temporary buffer prior to parsing, or to check the window range on every byte during header parsing. We may decide to revisit this decision in the future since this is just a gut instinct decision and has not actually been proven out by experimental testing. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Unmap individual windows rather than entire files.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | To support multiple windows per packfile we need to unmap only one window at a time from that packfile, leaving any other windows in place and available for reference. We treat all windows from all packfiles equally; the least recently used, not-in-use window across all packfiles will always be closed first. If we have unmapped all windows in a packfile then we can also close the packfile's file descriptor as its possible we won't need to map any window from that file in the near future. This decision about when to close the pack file descriptor may need to be revisited in the future after additional testing on several different platforms can be performed. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Document why header parsing won't exceed a window.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | When we parse the object header or the delta base reference we don't bother to loop over use_pack() calls. The reason we don't need to bother with calling use_pack for each byte accessed is that use_pack will always promise us at least 20 bytes (really the hash size) after the offset. This promise from use_pack simplifies a lot of code in the header parsing logic, as well as helps out the zlib library by ensuring there's always some data for it to consume during an inflate call. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Loop over pack_windows when inflating/accessing data.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | | | When multiple mmaps start getting used for all pack file access it is not possible to get all data associated with a specific object in one contiguous memory region. This limitation prevents simply passing a single address and length to SHA1_Update or to inflate. Instead we need to loop until we have processed all data of interest. As we loop over the data we are always interested in reusing the same window 'cursor', as the prior window will no longer be of any use to us. This allows the use_pack() call to automatically decrement the use count of the prior window before setting up access for us to the next window. Within each loop we need to make use of the available length output parameter of use_pack() to tell us how many bytes are available in the current memory region, as we cannot tell otherwise. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Replace use_packed_git with window cursors.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Part of the implementation concept of the sliding mmap window for pack access is to permit multiple windows per pack to be mapped independently. Since the inuse_cnt is associated with the mmap and not with the file, this value is in struct pack_window and needs to be incremented/decremented for each pack_window accessed by any code. To faciliate that implementation we need to replace all uses of use_packed_git() and unuse_packed_git() with a different API that follows struct pack_window objects rather than struct packed_git. The way this works is when we need to start accessing a pack for the first time we should setup a new window 'cursor' by declaring a local and setting it to NULL: struct pack_windows *w_curs = NULL; To obtain the memory region which contains a specific section of the pack file we invoke use_pack(), supplying the address of our current window cursor: unsigned int len; unsigned char *addr = use_pack(p, &w_curs, offset, &len); the returned address `addr` will be the first byte at `offset` within the pack file. The optional variable len will also be updated with the number of bytes remaining following the address. Multiple calls to use_pack() with the same window cursor will update the window cursor, moving it from one window to another when necessary. In this way each window cursor variable maintains only one struct pack_window inuse at a time. Finally before exiting the scope which originally declared the window cursor we must invoke unuse_pack() to unuse the current window (which may be different from the one that was first obtained from use_pack): unuse_pack(&w_curs); This implementation is still not complete with regards to multiple windows, as only one window per pack file is supported right now. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Refactor how we open pack files to prepare for multiple windows.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | To efficiently support mmaping of multiple regions of the same pack file we want to keep the pack's file descriptor open while we are actively working with that pack. So we are now keeping that file descriptor in packed_git.pack_fd and closing it only after we unmap the last window. This is going to increase the number of file descriptors that are in use at once, however that will be bounded by the total number of pack files present and therefore should not be very high. It is a small tradeoff which we may need to revisit after some testing can be done on various repositories and systems. For code clarity we also want to seperate out the implementation of how we open a pack file from the implementation which locates a suitable window (or makes a new one) from the given pack file. Since this is a rather large delta I'm taking advantage of doing it now, in a fairly isolated change. When we open a pack file we need to examine the header and trailer without having a mmap in place, as we may only need to mmap the middle section of this particular pack. Consequently the verification code has been refactored to make use of the new read_or_die function. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Refactor packed_git to prepare for sliding mmap windows.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | | The idea behind the sliding mmap window pack reader implementation is to have multiple mmap regions active against the same pack file, thereby allowing the process to mmap in only the active/hot sections of the pack and reduce overall virtual address space usage. To implement this we need to refactor the mmap related data (pack_base, pack_use_cnt) out of struct packed_git and move them into a new struct pack_window. We are refactoring the code to support a single struct pack_window per packfile, thereby emulating the prior behavior of mmap'ing the entire pack file. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Introduce new config option for mmap limit.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | | Rather than hardcoding the maximum number of bytes which can be mmapped from pack files we should make this value configurable, allowing the end user to increase or decrease this limit on a per-repository basis depending on the size of the repository and the capabilities of their operating system. In general users should not need to manually tune such a low-level setting within the core code, but being able to artifically limit the number of bytes which we can mmap at once from pack files will make it easier to craft test cases for the new mmap sliding window implementation. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Replace unpack_entry_gently with unpack_entry.Shawn O. Pearce2006-12-29
| | | | | | | | | | | | | | | | The unpack_entry_gently function currently has only two callers: the delta base resolution in sha1_file.c and the main loop of pack-check.c. Both of these must change to using unpack_entry directly when we implement sliding window mmap logic, so I'm doing it earlier to help break down the change set. This may cause a slight performance decrease for delta base resolution as well as for pack-check.c's verify_packfile(), as the pack use counter will be incremented and decremented for every object that is unpacked. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* clarify some error messages wrt unknown object typesNicolas Pitre2006-12-20
| | | | | | | | | If ever new object types are added for future extensions then better have current git version report them as "unknown" instead of "corrupted". Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* sha1_object_info(): be consistent with read_sha1_file()Johannes Schindelin2006-11-27
| | | | | | | | | | | | | | | | | | | | We used to try loose objects first with sha1_object_info(), but packed objects first with read_sha1_file(). Now, prefer packed objects over loose ones with sha1_object_info(), too. Usually the old behaviour would pose no problem, but when you tried to fix a fscked up repository by inserting a known-good pack, git cat-file $(git cat-file -t <sha1>) <sha1> could fail, even when git cat-file blob <sha1> would _not_ fail. Worse, a repack would fail, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Merge branch 'maint'Junio C Hamano2006-11-09
|\ | | | | | | | | * maint: Nicer error messages in case saving an object to db goes wrong
| * Nicer error messages in case saving an object to db goes wrongPetr Baudis2006-11-09
| | | | | | | | | | | | | | | | | | Currently the error e.g. when pushing to a read-only repository is quite confusing, this attempts to clean it up, unifies error reporting between various object writers and uses error() on couple more places. Signed-off-by: Petr Baudis <pasky@suse.cz> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Teach receive-pack how to keep pack files based on object count.Shawn Pearce2006-11-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since keeping a pushed pack or exploding it into loose objects should be a local repository decision this teaches receive-pack to decide if it should call unpack-objects or index-pack --stdin --fix-thin based on the setting of receive.unpackLimit and the number of objects contained in the received pack. If the number of objects (hdr_entries) in the received pack is below the value of receive.unpackLimit (which is 5000 by default) then we unpack-objects as we have in the past. If the hdr_entries >= receive.unpackLimit then we call index-pack and ask it to include our pid and hostname in the .keep file to make it easier to identify why a given pack has been kept in the repository. Currently this leaves every received pack as a kept pack. We really don't want that as received packs will tend to be small. Instead we want to delete the .keep file automatically after all refs have been updated. That is being left as room for future improvement. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Merge branch 'master' into np/index-packJunio C Hamano2006-11-03
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * master: (90 commits) gitweb: Better support for non-CSS aware web browsers gitweb: Output also empty patches in "commitdiff" view gitweb: Use git-for-each-ref to generate list of heads and/or tags for-each-ref: "creator" and "creatordate" fields Add --global option to git-repo-config. pack-refs: Store the full name of the ref even when packing only tags. git-clone documentation didn't mention --origin as equivalent of -o Minor grammar fixes for git-diff-index.txt link_temp_to_file: call adjust_shared_perm() only when we created the directory Remove uneccessarily similar printf() from print_ref_list() in builtin-branch pack-objects doesn't create random pack names branch: work in subdirectories. gitweb: Use 's' regexp modifier to secure against filenames with LF gitweb: Secure against commit-ish/tree-ish with the same name as path gitweb: esc_html() author in blame git-svnimport: support for partial imports link_temp_to_file: don't leave the path truncated on adjust_shared_perm failure Move deny_non_fast_forwards handling completely into receive-pack. revision traversal: --unpacked does not limit commit list anymore. Continue traversal when rev-list --unpacked finds a packed commit. ...
| * \ Merge branch 'maint'Junio C Hamano2006-11-02
| |\ \ | | |/ | | | | | | | | | | | | | | | * maint: git-clone documentation didn't mention --origin as equivalent of -o Minor grammar fixes for git-diff-index.txt link_temp_to_file: call adjust_shared_perm() only when we created the directory
| | * link_temp_to_file: call adjust_shared_perm() only when we created the directoryJohannes Schindelin2006-11-02
| | |
| * | Merge branch 'maint'Junio C Hamano2006-11-01
| |\ \ | | |/ | | | | | | | | | | | | * maint: pack-objects doesn't create random pack names link_temp_to_file: don't leave the path truncated on adjust_shared_perm failure
| | * link_temp_to_file: don't leave the path truncated on adjust_shared_perm failureJunio C Hamano2006-10-31
| | | | | | | | | | | | Signed-off-by: Junio C Hamano <junkio@cox.net>
* | | Allow short pack names to git-pack-objects --unpacked=.Shawn Pearce2006-10-29
|/ / | | | | | | | | | | | | | | | | | | | | This allows us to pass just the file name of a pack rather than the complete path when we want pack-objects to consider its contents as though they were loose objects. This can be helpful if $GIT_OBJECT_DIRECTORY contains shell metacharacters which make it cumbersome to pass complete paths safely in a shell script. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Merge branch 'np/pack'Junio C Hamano2006-10-22
|\ \ | |/ |/| | | | | | | | | | | | | | | | | | | | | | | | | | | * np/pack: add the capability for index-pack to read from a stream index-pack: compare only the first 20-bytes of the key. git-repack: repo.usedeltabaseoffset pack-objects: document --delta-base-offset option allow delta data reuse even if base object is a preferred base zap a debug remnant let the GIT native protocol use offsets to delta base when possible make pack data reuse compatible with both delta types make git-pack-objects able to create deltas with offset to base teach git-index-pack about deltas with offset to base teach git-unpack-objects about deltas with offset to base introduce delta objects with offset to base
| * make pack data reuse compatible with both delta typesNicolas Pitre2006-09-27
| | | | | | | | | | | | | | | | | | | | | | This is the missing part to git-pack-objects allowing it to reuse delta data to/from any of the two delta types. It can reuse delta from any type, and it outputs base offsets when --allow-delta-base-offset is provided and the base is also included in the pack. Otherwise it outputs base sha1 references just like it always did. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
| * introduce delta objects with offset to baseNicolas Pitre2006-09-27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds a new object, namely OBJ_OFS_DELTA, renames OBJ_DELTA to OBJ_REF_DELTA to better make the distinction between those two delta objects, and adds support for the handling of those new delta objects in sha1_file.c only. The OBJ_OFS_DELTA contains a relative offset from the delta object's position in a pack instead of the 20-byte SHA1 reference to identify the base object. Since the base is likely to be not so far away, the relative offset is more likely to have a smaller encoding on average than an absolute offset. And for those delta objects the base must always be stored first because there is no way to know the distance of later objects when streaming a pack. Hence this relative offset is always meant to be negative. The offset encoding is slightly denser than the one used for object size -- credits to <linux@horizon.com> (whoever this is) for bringing it to my attention. This allows for pack size reduction between 3.2% (Linux-2.6) to over 5% (linux-historic). Runtime pack access should be faster too since delta replay does skip a search in the pack index for each delta in a chain. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | reduce delta head inflated sizeNicolas Pitre2006-10-18
| | | | | | | | | | | | | | | | | | | | Supposing that both the base and result sizes were both full size 64-bit values, their encoding would occupy only 9.2 bytes each. Therefore inflating 64 bytes is way overkill. Limit it to 20 bytes instead which should be plenty enough for a couple years to come. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Replace open-coded version of hash_sha1_file()Rene Scharfe2006-10-15
| | | | | | | | | | Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Make write_sha1_file_prepare() voidRene Scharfe2006-10-15
| | | | | | | | | | | | | | | | Move file name generation from write_sha1_file_prepare() to the one caller that cares and make it a void function. Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Make write_sha1_file_prepare() staticRene Scharfe2006-10-14
| | | | | | | | | | | | | | There are no callers of write_sha1_file_prepare() left outside of sha1_file.c, so make it static. Signed-off-by: Junio C Hamano <junkio@cox.net>
* | Add hash_sha1_file()Rene Scharfe2006-10-14
|/ | | | | | | | | | Most callers of write_sha1_file_prepare() are only interested in the resulting hash but don't care about the returned file name or the header. This patch adds a simple wrapper named hash_sha1_file() which does just that, and converts potential callers. Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx> Signed-off-by: Junio C Hamano <junkio@cox.net>
* many cleanups to sha1_file.cNicolas Pitre2006-09-23
| | | | | | | | | | | | | | | | | | | | | | Those cleanups are mainly to set the table for the support of deltas with base objects referenced by offsets instead of sha1. This means that many pack lookup functions are converted to take a pack/offset tuple instead of a sha1. This eliminates many struct pack_entry usages since this structure carried redundent information in many cases, and it increased stack footprint needlessly for a couple recursively called functions that used to declare a local copy of it for every recursion loop. In the process, packed_object_info_detail() has been reorganized as well so to look much saner and more amenable to deltas with offset support. Finally the appropriate adjustments have been made to functions that depend on the above changes. But there is no functionality changes yet simply some code refactoring at this point. Signed-off-by: Nicolas Pitre <nico@cam.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Make hexval() available to others.Junio C Hamano2006-09-20
| | | | | | | builtin-mailinfo.c has its own hexval implementaiton but it can share the table-lookup one recently implemented in sha1_file.c Signed-off-by: Junio C Hamano <junkio@cox.net>
* Merge branch 'jc/pack'Junio C Hamano2006-09-17
|\ | | | | | | | | | | | | | | | | * jc/pack: pack-objects: document --revs, --unpacked and --all. pack-objects --unpacked=<existing pack> option. pack-objects: further work on internal rev-list logic. pack-objects: run rev-list equivalent internally. Separate object listing routines out of rev-list
| * pack-objects --unpacked=<existing pack> option.Junio C Hamano2006-09-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Incremental repack without -a essentially boils down to: rev-list --objects --unpacked --all | pack-objects $new_pack which picks up all loose objects that are still live and creates a new pack. This implements --unpacked=<existing pack> option to tell the revision walking machinery to pretend as if objects in such a pack are unpacked for the purpose of object listing. With this, we could say: rev-list --objects --unpacked=$active_pack --all | pack-objects $new_pack instead, to mean "all live loose objects but pretend as if objects that are in this pack are also unpacked". The newly created pack would be perfect for updating $active_pack by replacing it. Since pack-objects now knows how to do the rev-list's work itself internally, you can also write the above example by: pack-objects --unpacked=$active_pack --all $new_pack </dev/null Signed-off-by: Junio C Hamano <junkio@cox.net>
* | get_sha1_hex() micro-optimizationJunio C Hamano2006-09-09
|/ | | | | | | | | The function appeared high on a gprof output for a rev-list run of a non-trivial size, and it was an obvious low-hanging fruit. The code is from Linus. Signed-off-by: Junio C Hamano <junkio@cox.net>
* more lightweight revalidation while reusing deflated stream in packingJunio C Hamano2006-09-03
| | | | | | | | | | | | | | When copying from an existing pack and when copying from a loose object with new style header, the code makes sure that the piece we are going to copy out inflates well and inflate() consumes the data in full while doing so. The check to see if the xdelta really apply is quite expensive as you described, because you would need to have the image of the base object which can be represented as a delta against something else. Signed-off-by: Junio C Hamano <junkio@cox.net>
* Replace uses of strdup with xstrdup.Shawn Pearce2006-09-02
| | | | | | | | | | | | | | | | Like xmalloc and xrealloc xstrdup dies with a useful message if the native strdup() implementation returns NULL rather than a valid pointer. I just tried to use xstrdup in new code and found it to be missing. However I expected it to be present as xmalloc and xrealloc are already commonly used throughout the code. [jc: removed the part that deals with last_XXX, which I am finding more and more dubious these days.] Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>
* consolidate two copies of new style object header parsing code.Junio C Hamano2006-09-01
| | | | | | | | | | | | Also while we are at it, remove redundant typename[] array from unpack_sha1_header. The only reason it is different from the type_names[] array in object.c module is that this code cares about the subset of object types that are valid in a loose object, so prepare a separate array of boolean that tells us which types are valid, and share the name translation with the others. Signed-off-by: Junio C Hamano <junkio@cox.net>
* Constness tightening for move/link_temp_to_file()Junio C Hamano2006-09-01
| | | | Signed-off-by: Junio C Hamano <junkio@cox.net>
* Use xmalloc instead of mallocJonas Fonseca2006-08-31
| | | | | Signed-off-by: Jonas Fonseca <fonseca@diku.dk> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Use xrealloc instead of reallocJonas Fonseca2006-08-26
| | | | | | | | | Change places that use realloc, without a proper error path, to instead use xrealloc. Drop an erroneous error path in the daemon code that used errno in the die message in favour of the simpler xrealloc. Signed-off-by: Jonas Fonseca <fonseca@diku.dk> Signed-off-by: Junio C Hamano <junkio@cox.net>
* Convert unpack_entry_gently and friends to use offsets.Shawn Pearce2006-08-26
| | | | | | | | | | | | | Change unpack_entry_gently and its helper functions to use offsets rather than addresses and left counts to supply pack position information. In most cases this makes the code easier to follow, and it reduces the number of local variables in a few functions. It also better prepares this code for mapping partial segments of packs and altering what regions of a pack are mapped while unpacking an entry. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> Signed-off-by: Junio C Hamano <junkio@cox.net>