aboutsummaryrefslogtreecommitdiff
path: root/convert.c
Commit message (Collapse)AuthorAge
* Merge branch 'tb/check-crlf-for-safe-crlf'Junio C Hamano2017-12-27
|\ | | | | | | | | | | | | | | | | The "safe crlf" check incorrectly triggered for contents that does not use CRLF as line endings, which has been corrected. * tb/check-crlf-for-safe-crlf: t0027: Adapt the new MIX tests to Windows convert: tighten the safe autocrlf handling
| * convert: tighten the safe autocrlf handlingTorsten Bögershausen2017-11-27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a text file had been commited with CRLF and the file is commited again, the CRLF are kept if .gitattributs has "text=auto". This is done by analyzing the content of the blob stored in the index: If a '\r' is found, Git assumes that the blob was commited with CRLF. The simple search for a '\r' does not always work as expected: A file is encoded in UTF-16 with CRLF and commited. Git treats it as binary. Now the content is converted into UTF-8. At the next commit Git treats the file as text, the CRLF should be converted into LF, but isn't. Replace has_cr_in_index() with has_crlf_in_index(). When no '\r' is found, 0 is returned directly, this is the most common case. If a '\r' is found, the content is analyzed more deeply. Reported-By: Ashish Negi <ashishnegi33@gmail.com> Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * Merge branch 'ma/ts-cleanups' into maintJunio C Hamano2017-10-23
| |\ | | | | | | | | | | | | | | | | | | | | | | | | | | | Assorted bugfixes and clean-ups. * ma/ts-cleanups: ThreadSanitizer: add suppressions strbuf_setlen: don't write to strbuf_slopbuf pack-objects: take lock before accessing `remaining` convert: always initialize attr_action in convert_attrs
| * \ Merge branch 'tb/apply-with-crlf' into maintJunio C Hamano2017-09-10
| |\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "git apply" that is used as a better "patch -p1" failed to apply a taken from a file with CRLF line endings to a file with CRLF line endings. The root cause was because it misused convert_to_git() that tried to do "safe-crlf" processing by looking at the index entry at the same path, which is a nonsense---in that mode, "apply" is not working on the data in (or derived from) the index at all. This has been fixed. * tb/apply-with-crlf: apply: file commited with CRLF should roundtrip diff and apply convert: add SAFE_CRLF_KEEP_CRLF
* | | | consistently use "fallthrough" comments in switchesJeff King2017-09-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Gcc 7 adds -Wimplicit-fallthrough, which can warn when a switch case falls through to the next case. The general idea is that the compiler can't tell if this was intentional or not, so you should annotate any intentional fall-throughs as such, leaving it to complain about any unannotated ones. There's a GNU __attribute__ which can be used for annotation, but of course we'd have to #ifdef it away on non-gcc compilers. Gcc will also recognize specially-formatted comments, which matches our current practice. Let's extend that practice to all of the unannotated sites (which I did look over and verify that they were behaving as intended). Ideally in each case we'd actually give some reasons in the comment about why we're falling through, or what we're falling through to. And gcc does support that with -Wimplicit-fallthrough=2, which relaxes the comment pattern matching to anything that contains "fallthrough" (or a variety of spelling variants). However, this isn't the default for -Wimplicit-fallthrough, nor for -Wextra. In the name of simplicity, it's probably better for us to support the default level, which requires "fallthrough" to be the only thing in the comment (modulo some window dressing like "else" and some punctuation; see the gcc manual for the complete set of patterns). This patch suppresses all warnings due to -Wimplicit-fallthrough. We might eventually want to add that to the DEVELOPER Makefile knob, but we should probably wait until gcc 7 is more widely adopted (since earlier versions will complain about the unknown warning type). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | | Merge branch 'rs/strbuf-leakfix'Junio C Hamano2017-09-19
|\ \ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many leaks of strbuf have been fixed. * rs/strbuf-leakfix: (34 commits) wt-status: release strbuf after use in wt_longstatus_print_tracking() wt-status: release strbuf after use in read_rebase_todolist() vcs-svn: release strbuf after use in end_revision() utf8: release strbuf on error return in strbuf_utf8_replace() userdiff: release strbuf after use in userdiff_get_textconv() transport-helper: release strbuf after use in process_connect_service() sequencer: release strbuf after use in save_head() shortlog: release strbuf after use in insert_one_record() sha1_file: release strbuf on error return in index_path() send-pack: release strbuf on error return in send_pack() remote: release strbuf after use in set_url() remote: release strbuf after use in migrate_file() remote: release strbuf after use in read_remote_branches() refs: release strbuf on error return in write_pseudoref() notes: release strbuf after use in notes_copy_from_stdin() merge: release strbuf after use in write_merge_heads() merge: release strbuf after use in save_state() mailinfo: release strbuf on error return in handle_boundary() mailinfo: release strbuf after use in handle_from() help: release strbuf on error return in exec_woman_emacs() ...
| * | | | convert: release strbuf on error return in filter_buffer_or_fd()Rene Scharfe2017-09-07
| | | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | | | Merge branch 'ma/ts-cleanups'Junio C Hamano2017-09-10
|\ \ \ \ \ | |/ / / / |/| | | / | | |_|/ | |/| | | | | | | | | | | | | | | | | | | | | | Assorted bugfixes and clean-ups. * ma/ts-cleanups: ThreadSanitizer: add suppressions strbuf_setlen: don't write to strbuf_slopbuf pack-objects: take lock before accessing `remaining` convert: always initialize attr_action in convert_attrs
| * | | convert: always initialize attr_action in convert_attrsMartin Ågren2017-08-23
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | convert_attrs contains an "if-else". In the "if", we set attr_action twice, and the first assignment has no effect. In the "else", we do not set it at all. Since git_check_attr always returns the same value, we'll always end up in the "if", so there is no problem right now. But convert_attrs is obviously trying not to rely on such an implementation-detail of another component. Make the initialization of attr_action after the if-else. Remove the earlier assignments. Suggested-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Martin Ågren <martin.agren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | | Merge branch 'tb/apply-with-crlf'Junio C Hamano2017-08-26
|\ \ \ \ | | |_|/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | "git apply" that is used as a better "patch -p1" failed to apply a taken from a file with CRLF line endings to a file with CRLF line endings. The root cause was because it misused convert_to_git() that tried to do "safe-crlf" processing by looking at the index entry at the same path, which is a nonsense---in that mode, "apply" is not working on the data in (or derived from) the index at all. This has been fixed. * tb/apply-with-crlf: apply: file commited with CRLF should roundtrip diff and apply convert: add SAFE_CRLF_KEEP_CRLF
| * | | convert: add SAFE_CRLF_KEEP_CRLFTorsten Bögershausen2017-08-16
| |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When convert_to_git() is called, the caller may want to keep CRLF to be kept as CRLF (and not converted into LF). This will be used in the next commit, when apply works with files that have CRLF and patches are applied onto these files. Add the new value "SAFE_CRLF_KEEP_CRLF" to safe_crlf. Prepare convert_to_git() to be able to run the clean filter, skip the CRLF conversion and run the ident filter. Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'jt/subprocess-handshake'Junio C Hamano2017-08-11
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | Code cleanup. * jt/subprocess-handshake: sub-process: refactor handshake to common function Documentation: migrate sub-process docs to header
| * | sub-process: refactor handshake to common functionJonathan Tan2017-07-26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor, into a common function, the version and capability negotiation done when invoking a long-running process as a clean or smudge filter. This will be useful for other Git code that needs to interact similarly with a long-running process. As you can see in the change to t0021, this commit changes the error message reported when the long-running process does not introduce itself with the expected "server"-terminated line. Originally, the error message reports that the filter "does not support filter protocol version 2", differentiating between the old single-file filter protocol and the new multi-file filter protocol - I have updated it to something more generic and useful. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | Merge branch 'ls/filter-process-delayed' into jt/subprocess-handshakeJunio C Hamano2017-07-26
| |\ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | * ls/filter-process-delayed: convert: add "status=delayed" to filter process protocol convert: refactor capabilities negotiation convert: move multiple file filter error handling to separate function convert: put the flags field before the flag itself for consistent style t0021: write "OUT <size>" only on success t0021: make debug log file name configurable t0021: keep filter log files on comparison
* | | Merge branch 'sb/hashmap-cleanup'Junio C Hamano2017-08-11
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Many uses of comparision callback function the hashmap API uses cast the callback function type when registering it to hashmap_init(), which defeats the compile time type checking when the callback interface changes (e.g. gaining more parameters). The callback implementations have been updated to take "void *" pointers and cast them to the type they expect instead. * sb/hashmap-cleanup: t/helper/test-hashmap: use custom data instead of duplicate cmp functions name-hash.c: drop hashmap_cmp_fn cast submodule-config.c: drop hashmap_cmp_fn cast remote.c: drop hashmap_cmp_fn cast patch-ids.c: drop hashmap_cmp_fn cast convert/sub-process: drop cast to hashmap_cmp_fn config.c: drop hashmap_cmp_fn cast builtin/describe: drop hashmap_cmp_fn cast builtin/difftool.c: drop hashmap_cmp_fn cast attr.c: drop hashmap_cmp_fn cast
| * | | convert/sub-process: drop cast to hashmap_cmp_fnStefan Beller2017-07-05
| |/ / | | | | | | | | | | | | Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | Merge branch 'ls/filter-process-delayed'Junio C Hamano2017-08-11
|\ \ \ | |/ / |/| / | |/ | | | | | | | | | | | | | | | | | | | | The filter-process interface learned to allow a process with long latency give a "delayed" response. * ls/filter-process-delayed: convert: add "status=delayed" to filter process protocol convert: refactor capabilities negotiation convert: move multiple file filter error handling to separate function convert: put the flags field before the flag itself for consistent style t0021: write "OUT <size>" only on success t0021: make debug log file name configurable t0021: keep filter log files on comparison
| * convert: add "status=delayed" to filter process protocolLars Schneider2017-06-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some `clean` / `smudge` filters may require a significant amount of time to process a single blob (e.g. the Git LFS smudge filter might perform network requests). During this process the Git checkout operation is blocked and Git needs to wait until the filter is done to continue with the checkout. Teach the filter process protocol, introduced in edcc8581 ("convert: add filter.<driver>.process option", 2016-10-16), to accept the status "delayed" as response to a filter request. Upon this response Git continues with the checkout operation. After the checkout operation Git calls "finish_delayed_checkout" which queries the filter for remaining blobs. If the filter is still working on the completion, then the filter is expected to block. If the filter has completed all remaining blobs then an empty response is expected. Git has a multiple code paths that checkout a blob. Support delayed checkouts only in `clone` (in unpack-trees.c) and `checkout` operations for now. The optimization is most effective in these code paths as all files of the tree are processed. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * convert: refactor capabilities negotiationLars Schneider2017-06-30
| | | | | | | | | | | | | | | | | | | | | | | | The code to negotiate long running filter capabilities was very repetitive for new capabilities. Replace the repetitive conditional statements with a table-driven approach. This is useful for the subsequent patch 'convert: add "status=delayed" to filter process protocol'. Suggested-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * convert: move multiple file filter error handling to separate functionLars Schneider2017-06-29
| | | | | | | | | | | | | | | | | | | | | | Refactoring the filter error handling is useful for the subsequent patch 'convert: add "status=delayed" to filter process protocol'. In addition, replace the parentheses around the empty "if" block with a single semicolon to adhere to the Git style guide. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * convert: put the flags field before the flag itself for consistent styleLars Schneider2017-06-29
| | | | | | | | | | | | Suggested-by: Jeff King <peff@peff.net> Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | hashmap.h: compare function has access to a data fieldStefan Beller2017-06-30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When using the hashmap a common need is to have access to caller provided data in the compare function. A couple of times we abuse the keydata field to pass in the data needed. This happens for example in patch-ids.c. This patch changes the function signature of the compare function to have one more void pointer available. The pointer given for each invocation of the compare function must be defined in the init function of the hashmap and is just passed through. Documentation of this new feature is deferred to a later patch. This is a rather mechanical conversion, just adding the new pass-through parameter. However while at it improve the naming of the fields of all compare functions used by hashmaps by ensuring unused parameters are prefixed with 'unused_' and naming the parameters what they are (instead of 'unused' make it 'unused_keydata'). Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'bw/config-h'Junio C Hamano2017-06-24
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix configuration codepath to pay proper attention to commondir that is used in multi-worktree situation, and isolate config API into its own header file. * bw/config-h: config: don't implicitly use gitdir or commondir config: respect commondir setup: teach discover_git_directory to respect the commondir config: don't include config.h by default config: remove git_config_iter config: create config.h
| * | config: don't include config.h by defaultBrandon Williams2017-06-15
| | | | | | | | | | | | | | | | | | | | | | | | Stop including config.h by default in cache.h. Instead only include config.h in those files which require use of the config system. Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | convert: convert renormalize_buffer to take an indexBrandon Williams2017-06-13
| | | | | | | | | | | | | | | Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | convert: convert convert_to_git to take an indexBrandon Williams2017-06-13
| | | | | | | | | | | | | | | Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | convert: convert convert_to_git_filter_fd to take an indexBrandon Williams2017-06-13
| | | | | | | | | | | | | | | Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | convert: convert crlf_to_git to take an indexBrandon Williams2017-06-13
| | | | | | | | | | | | | | | Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | | convert: convert get_cached_convert_stats_ascii to take an indexBrandon Williams2017-06-13
| |/ |/| | | | | | | Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: update subprocess_read_status() to not die on EOFBen Peart2017-05-15
| | | | | | | | | | | | | | | | | | | | | | | | Enable sub-processes to gracefully handle when the process dies by updating subprocess_read_status to return an error on EOF instead of dying. Update apply_multi_file_filter to take advantage of the revised subprocess_read_status. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | sub-process: move sub-process functions into separate filesBen Peart2017-05-15
| | | | | | | | | | | | | | | | Move the sub-proces functions into sub-process.h/c. Add documentation for the new module in Documentation/technical/api-sub-process.txt Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: rename reusable sub-process functionsBen Peart2017-05-15
| | | | | | | | | | | | | | | | Do a mechanical rename of the functions that will become the reusable sub-process module. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: update generic functions to only use generic data structuresBen Peart2017-05-15
| | | | | | | | | | | | | | | | | | Update all functions that are going to be moved into a reusable module so that they only work with the reusable data structures. Move code that is specific to the filter out into the filter specific functions. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: separate generic structures and variables from the filter specific onesBen Peart2017-05-15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | To enable future reuse of the filter.<driver>.process infrastructure, split the cmd2process structure into two separate parts. subprocess_entry will now contain the generic data required to manage the creation and tracking of the child process in a hashmap. cmd2process is a filter protocol specific structure that is used to track the negotiated capabilities of the filter. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: split start_multi_file_filter() into two separate functionsBen Peart2017-05-15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To enable future reuse of the filter.<driver>.process infrastructure, split start_multi_file_filter() into two separate parts. start_multi_file_filter() will now only contain the generic logic to manage the creation and tracking of the child process in a hashmap. start_multi_file_filter_fn() is a protocol specific initialization function that will negotiate the multi-file-filter interface version and capabilities. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: move packet_write_line() into pkt-line as packet_writel()Ben Peart2017-05-08
| | | | | | | | | | | | | | | | | | Add packet_writel() which writes multiple lines in a single call and then calls packet_flush_gently(). Update convert.c to use the new packet_writel() function from pkt-line. Signed-off-by: Ben Peart <benpeart@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | convert: remove erroneous tests for errno == EPIPEBen Peart2017-05-08
|/ | | | | | | | | | | start_multi_file_filter() and apply_multi_file_filter() currently test for errno == EPIPE but treating EPIPE as an error is already happening from one of the packet_write() functions. Signed-off-by: Ben Peart <benpeart@microsoft.com> Found/Fixed-by: Jeff King <peff@peff.net> Acked-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* attr: convert git_check_attrs() callers to use the new APIJunio C Hamano2017-02-01
| | | | | | | | | | | | | | | The remaining callers are all simple "I have N attributes I am interested in. I'll ask about them with various paths one by one". After this step, no caller to git_check_attrs() remains. After removing it, we can extend "struct attr_check" struct with data that can be used in optimizing the query for the specific N attributes it contains. Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* attr: rename function and struct related to checking attributesJunio C Hamano2017-02-01
| | | | | | | | | | | | | | | | The traditional API to check attributes is to prepare an N-element array of "struct git_attr_check" and pass N and the array to the function "git_check_attr()" as arguments. In preparation to revamp the API to pass a single structure, in which these N elements are held, rename the type used for these individual array elements to "struct attr_check_item" and rename the function to "git_check_attrs()". Signed-off-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Brandon Williams <bmwill@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* Merge branch 'jc/renormalize-merge-kill-safer-crlf'Junio C Hamano2016-12-19
|\ | | | | | | | | | | | | | | | | | | Fix a corner case in merge-recursive regression that crept in during 2.10 development cycle. * jc/renormalize-merge-kill-safer-crlf: convert: git cherry-pick -Xrenormalize did not work merge-recursive: handle NULL in add_cacheinfo() correctly cherry-pick: demonstrate a segmentation fault
| * convert: git cherry-pick -Xrenormalize did not workTorsten Bögershausen2016-12-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Working with a repo that used to be all CRLF. At some point it was changed to all LF, with `text=auto` in .gitattributes. Trying to cherry-pick a commit from before the switchover fails: $ git cherry-pick -Xrenormalize <commit> fatal: CRLF would be replaced by LF in [path] Commit 65237284 "unify the "auto" handling of CRLF" introduced a regression: Whenever crlf_action is CRLF_TEXT_XXX and not CRLF_AUTO_XXX, SAFE_CRLF_RENORMALIZE was feed into check_safe_crlf(). This is wrong because here everything else than SAFE_CRLF_WARN is treated as SAFE_CRLF_FAIL. Call check_safe_crlf() only if checksafe is SAFE_CRLF_WARN or SAFE_CRLF_FAIL. Reported-by: Eevee (Lexy Munroe) <eevee@veekun.com> Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | Merge branch 'ls/filter-process'Junio C Hamano2016-10-31
|\ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The smudge/clean filter API expect an external process is spawned to filter the contents for each path that has a filter defined. A new type of "process" filter API has been added to allow the first request to run the filter for a path to spawn a single process, and all filtering need is served by this single process for multiple paths, reducing the process creation overhead. * ls/filter-process: contrib/long-running-filter: add long running filter example convert: add filter.<driver>.process option convert: prepare filter.<driver>.process option convert: make apply_filter() adhere to standard Git error handling pkt-line: add functions to read/write flush terminated packet streams pkt-line: add packet_write_gently() pkt-line: add packet_flush_gently() pkt-line: add packet_write_fmt_gently() pkt-line: extract set_packet_header() pkt-line: rename packet_write() to packet_write_fmt() run-command: add clean_on_exit_handler run-command: move check_pipe() from write_or_die to run_command convert: modernize tests convert: quote filter names in error messages
| * | convert: add filter.<driver>.process optionLars Schneider2016-10-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Git's clean/smudge mechanism invokes an external filter process for every single blob that is affected by a filter. If Git filters a lot of blobs then the startup time of the external filter processes can become a significant part of the overall Git execution time. In a preliminary performance test this developer used a clean/smudge filter written in golang to filter 12,000 files. This process took 364s with the existing filter mechanism and 5s with the new mechanism. See details here: https://github.com/github/git-lfs/pull/1382 This patch adds the `filter.<driver>.process` string option which, if used, keeps the external filter process running and processes all blobs with the packet format (pkt-line) based protocol over standard input and standard output. The full protocol is explained in detail in `Documentation/gitattributes.txt`. A few key decisions: * The long running filter process is referred to as filter protocol version 2 because the existing single shot filter invocation is considered version 1. * Git sends a welcome message and expects a response right after the external filter process has started. This ensures that Git will not hang if a version 1 filter is incorrectly used with the filter.<driver>.process option for version 2 filters. In addition, Git can detect this kind of error and warn the user. * The status of a filter operation (e.g. "success" or "error) is set before the actual response and (if necessary!) re-set after the response. The advantage of this two step status response is that if the filter detects an error early, then the filter can communicate this and Git does not even need to create structures to read the response. * All status responses are pkt-line lists terminated with a flush packet. This allows us to send other status fields with the same protocol in the future. Helped-by: Martin-Louis Bright <mlbright@gmail.com> Reviewed-by: Jakub Narebski <jnareb@gmail.com> Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | convert: prepare filter.<driver>.process optionLars Schneider2016-10-17
| | | | | | | | | | | | | | | | | | | | | | | | Refactor the existing 'single shot filter mechanism' and prepare the new 'long running filter mechanism'. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | convert: make apply_filter() adhere to standard Git error handlingLars Schneider2016-10-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | apply_filter() returns a boolean that tells the caller if it "did convert or did not convert". The variable `ret` was used throughout the function to track errors whereas `1` denoted success and `0` failure. This is unusual for the Git source where `0` denotes success. Rename the variable and flip its value to make the function easier readable for Git developers. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
| * | convert: quote filter names in error messagesLars Schneider2016-10-17
| |/ | | | | | | | | | | | | | | Git filter driver commands with spaces (e.g. `filter.sh foo`) are hard to read in error messages. Quote them to improve the readability. Signed-off-by: Lars Schneider <larsxschneider@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* | i18n: convert mark error messages for translationVasco Almeida2016-10-17
|/ | | | | | | | | Mark error messages about CRLF for translation. Update test to reflect changes. Signed-off-by: Vasco Almeida <vascomalmeida@sapo.pt> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* convert: Correct NNO tests and missing `LF will be replaced by CRLF`Torsten Bögershausen2016-08-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a non-reversible CRLF conversion is done in "git add", a warning is printed on stderr (or Git dies, depending on checksafe) The function commit_chk_wrnNNO() in t0027 was written to test this, but did the wrong thing: Instead of looking at the warning from "git add", it looked at the warning from "git commit". This is racy because "git commit" may not have to do CRLF conversion at all if it can use the sha1 value from the index (which depends on whether "add" and "commit" run in a single second). Correct t0027 and replace the commit for each and every file with a commit of all files in one go. The function commit_chk_wrnNNO() should be renamed in a separate commit. Now that t0027 does the right thing, it detects a bug in covert.c: This sequence should generate the warning `LF will be replaced by CRLF`, but does not: $ git init $ git config core.autocrlf false $ printf "Line\r\n" >file $ git add file $ git commit -m "commit with CRLF" $ git config core.autocrlf true $ printf "Line\n" >file $ git add file "git add" calls crlf_to_git() in convert.c, which calls check_safe_crlf(). When has_cr_in_index(path) is true, crlf_to_git() returns too early and check_safe_crlf() is not called at all. Factor out the code which determines if "git checkout" converts LF->CRLF into will_convert_lf_to_crlf(). Update the logic around check_safe_crlf() and "simulate" the possible LF->CRLF conversion at "git checkout" with help of will_convert_lf_to_crlf(). Thanks to Jeff King <peff@peff.net> for analyzing t0027. Reported-By: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* convert: unify the "auto" handling of CRLFTorsten Bögershausen2016-07-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this change, $ echo "* text=auto" >.gitattributes $ echo "* eol=crlf" >>.gitattributes would have the same effect as $ echo "* text" >.gitattributes $ git config core.eol crlf Since the 'eol' attribute had higher priority than 'text=auto', this may corrupt binary files and is not what most users expect to happen. Make the 'eol' attribute to obey 'text=auto' and now $ echo "* text=auto" >.gitattributes $ echo "* eol=crlf" >>.gitattributes behaves the same as $ echo "* text=auto" >.gitattributes $ git config core.eol crlf In other words, $ echo "* text=auto eol=crlf" >.gitattributes has the same effect as $ git config core.autocrlf true and $ echo "* text=auto eol=lf" >.gitattributes has the same effect as $ git config core.autocrlf input Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
* convert.c: ident + core.autocrlf didn't workTorsten Bögershausen2016-04-25
| | | | | | | | | | | | | | | When the ident attributes is set, get_stream_filter() did not obey core.autocrlf=true, and the file was checked out with LF. Change the rule when a streaming filter can be used: - if an external filter is specified, don't use a stream filter. - if the worktree eol is CRLF and "auto" is active, don't use a stream filter. - Otherwise the stream filter can be used. Add test cases in t0027. Signed-off-by: Torsten Bögershausen <tboegi@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
generated by cgit v1.2.1 (git 2.18.0) at 2024-05-18 05:30:55 +0000