diff options
Diffstat (limited to 't/README')
| -rw-r--r-- | t/README | 279 |
1 files changed, 251 insertions, 28 deletions
@@ -50,6 +50,12 @@ prove and other harnesses come with a lot of useful options. The # Repeat until no more failures $ prove -j 15 --state=failed,save ./t[0-9]*.sh +You can give DEFAULT_TEST_TARGET=prove on the make command (or define it +in config.mak) to cause "make test" to run tests under prove. +GIT_PROVE_OPTS can be used to pass additional options, e.g. + + $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--timer --jobs 16' test + You can also run each test individually from command line, like this: $ sh ./t3010-ls-files-killed-modified.sh @@ -73,6 +79,10 @@ appropriately before running "make". --debug:: This may help the person who is developing a new test. It causes the command defined with test_debug to run. + The "trash" directory (used to store all temporary data + during testing) is not deleted even if there are no + failed tests so that you can inspect its contents after + the test finished. --immediate:: This causes the test to immediately exit upon the first @@ -92,6 +102,13 @@ appropriately before running "make". not see any output, this option implies --verbose. For convenience, it also implies --tee. + Note that valgrind is run with the option --leak-check=no, + as the git process is short-lived and some errors are not + interesting. In order to run a single command under the same + conditions manually, you should set GIT_VALGRIND to point to + the 't/valgrind/' directory and use the commands under + 't/valgrind/bin/'. + --tee:: In addition to printing the test output to the terminal, write it to files named 't/test-results/$TEST_NAME.out'. @@ -184,7 +201,7 @@ we are testing. If you create files under t/ directory (i.e. here) that is not the top-level test script, never name the file to match the above pattern. The Makefile here considers all such files as the -top-level test script and tries to run all of them. A care is +top-level test script and tries to run all of them. Care is especially needed if you are creating a common test library file, similar to test-lib.sh, because such a library file may not be suitable for standalone execution. @@ -259,14 +276,28 @@ Do: test ... That way all of the commands in your tests will succeed or fail. If - you must ignore the return value of something (e.g., the return - after unsetting a variable that was already unset is unportable) it's - best to indicate so explicitly with a semicolon: + you must ignore the return value of something, consider using a + helper function (e.g. use sane_unset instead of unset, in order + to avoid unportable return value for unsetting a variable that was + already unset), or prepending the command with test_might_fail or + test_must_fail. - unset HLAGH; - git merge hla && - git push gh && - test ... + - Check the test coverage for your tests. See the "Test coverage" + below. + + Don't blindly follow test coverage metrics; if a new function you added + doesn't have any coverage, then you're probably doing something wrong, + but having 100% coverage doesn't necessarily mean that you tested + everything. + + Tests that are likely to smoke out future regressions are better + than tests that just inflate the coverage metrics. + + - When a test checks for an absolute path that a git command generated, + construct the expected value using $(pwd) rather than $PWD, + $TEST_DIRECTORY, or $TRASH_DIRECTORY. It makes a difference on + Windows, where the shell (MSYS bash) mangles absolute path names. + For details, see the commit message of 4114156ae9. Don't: @@ -307,9 +338,21 @@ Keep in mind: Skipping tests -------------- -If you need to skip all the remaining tests you should set skip_all -and immediately call test_done. The string you give to skip_all will -be used as an explanation for why the test was skipped. for instance: +If you need to skip tests you should do so by using the three-arg form +of the test_* functions (see the "Test harness library" section +below), e.g.: + + test_expect_success PERL 'I need Perl' " + '$PERL_PATH' -e 'hlagh() if unf_unf()' + " + +The advantage of skipping tests like this is that platforms that don't +have the PERL and other optional dependencies get an indication of how +many tests they're missing. + +If the test code is too hairy for that (i.e. does a lot of setup work +outside test assertions) you can also skip all remaining tests by +setting skip_all and immediately call test_done: if ! test_have_prereq PERL then @@ -317,6 +360,9 @@ be used as an explanation for why the test was skipped. for instance: test_done fi +The string you give to skip_all will be used as an explanation for why +the test was skipped. + End with test_done ------------------ @@ -333,7 +379,7 @@ library for your script to use. - test_expect_success [<prereq>] <message> <script> - Usually takes two strings as parameter, and evaluates the + Usually takes two strings as parameters, and evaluates the <script>. If it yields success, test is considered successful. <message> should state what it is testing. @@ -344,12 +390,18 @@ library for your script to use. 'tree=$(git-write-tree)' If you supply three parameters the first will be taken to be a - prerequisite, see the test_set_prereq and test_have_prereq + prerequisite; see the test_set_prereq and test_have_prereq documentation below: test_expect_success TTY 'git --paginate rev-list uses a pager' \ ' ... ' + You can also supply a comma-separated list of prerequisites, in the + rare case where your test depends on more than one: + + test_expect_success PERL,PYTHON 'yo dawg' \ + ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' + - test_expect_failure [<prereq>] <message> <script> This is NOT the opposite of test_expect_success, but is used @@ -362,13 +414,6 @@ library for your script to use. Like test_expect_success this function can optionally use a three argument invocation with a prerequisite as the first argument. - - test_expect_code [<prereq>] <code> <message> <script> - - Analogous to test_expect_success, but pass the test if it exits - with a given exit <code> - - test_expect_code 1 'Merge with d/f conflicts' 'git merge "merge msg" B master' - - test_debug <script> This takes a single argument, <script>, and evaluates it only @@ -385,7 +430,7 @@ library for your script to use. - test_tick Make commit and tag names consistent by setting the author and - committer times to defined stated. Subsequent calls will + committer times to defined state. Subsequent calls will advance the times by a fixed amount. - test_commit <message> [<filename> [<contents>]] @@ -401,16 +446,17 @@ library for your script to use. Merges the given rev using the given message. Like test_commit, creates a tag and calls test_tick before committing. - - test_set_prereq SOME_PREREQ + - test_set_prereq <prereq> Set a test prerequisite to be used later with test_have_prereq. The - test-lib will set some prerequisites for you, e.g. PERL and PYTHON - which are derived from ./GIT-BUILD-OPTIONS (grep test_set_prereq - test-lib.sh for more). Others you can set yourself and use later - with either test_have_prereq directly, or the three argument - invocation of test_expect_success and test_expect_failure. + test-lib will set some prerequisites for you, see the + "Prerequisites" section below for a full list of these. - - test_have_prereq SOME PREREQ + Others you can set yourself and use later with either + test_have_prereq directly, or the three argument invocation of + test_expect_success and test_expect_failure. + + - test_have_prereq <prereq> Check if we have a prerequisite previously set with test_set_prereq. The most common use of this directly is to skip @@ -448,6 +494,15 @@ library for your script to use. 'Perl API' \ "$PERL_PATH" "$TEST_DIRECTORY"/t9700/test.pl + - test_expect_code <exit-code> <command> + + Run a command and ensure that it exits with the given exit code. + For example: + + test_expect_success 'Merge with d/f conflicts' ' + test_expect_code 1 git merge "merge msg" B master + ' + - test_must_fail <git-command> Run a git command and ensure it fails in a controlled way. Use @@ -467,6 +522,18 @@ library for your script to use. <expected> file. This behaves like "cmp" but produces more helpful output when the test is run with "-v" option. + - test_line_count (= | -lt | -ge | ...) <length> <file> + + Check whether a file has the length it is expected to. + + - test_path_is_file <path> [<diagnosis>] + test_path_is_dir <path> [<diagnosis>] + test_path_is_missing <path> [<diagnosis>] + + Check if the named path is a file, if the named path is a + directory, or if the named path does not exist, respectively, + and fail otherwise, showing the <diagnosis> text. + - test_when_finished <script> Prepend <script> to a list of commands to run to clean up @@ -481,6 +548,50 @@ library for your script to use. ... ' +Prerequisites +------------- + +These are the prerequisites that the test library predefines with +test_have_prereq. + +See the prereq argument to the test_* functions in the "Test harness +library" section above and the "test_have_prereq" function for how to +use these, and "test_set_prereq" for how to define your own. + + - PERL & PYTHON + + Git wasn't compiled with NO_PERL=YesPlease or + NO_PYTHON=YesPlease. Wrap any tests that need Perl or Python in + these. + + - POSIXPERM + + The filesystem supports POSIX style permission bits. + + - BSLASHPSPEC + + Backslashes in pathspec are not directory separators. This is not + set on Windows. See 6fd1106a for details. + + - EXECKEEPSPID + + The process retains the same pid across exec(2). See fb9a2bea for + details. + + - SYMLINKS + + The filesystem we're on supports symbolic links. E.g. a FAT + filesystem doesn't support these. See 704a3143 for details. + + - SANITY + + Test is not run by root user, and an attempt to write to an + unwritable file is expected to fail correctly. + + - LIBPCRE + + Git was compiled with USE_LIBPCRE=YesPlease. Wrap any tests + that use git-grep --perl-regexp or git-grep -P in these. Tips for Writing Tests ---------------------- @@ -508,3 +619,115 @@ the purpose of t0000-basic.sh, which is to isolate that level of validation in one place. Your test also ends up needing updating when such a change to the internal happens, so do _not_ do it and leave the low level of validation to t0000-basic.sh. + +Test coverage +------------- + +You can use the coverage tests to find code paths that are not being +used or properly exercised yet. + +To do that, run the coverage target at the top-level (not in the t/ +directory): + + make coverage + +That'll compile Git with GCC's coverage arguments, and generate a test +report with gcov after the tests finish. Running the coverage tests +can take a while, since running the tests in parallel is incompatible +with GCC's coverage mode. + +After the tests have run you can generate a list of untested +functions: + + make coverage-untested-functions + +You can also generate a detailed per-file HTML report using the +Devel::Cover module. To install it do: + + # On Debian or Ubuntu: + sudo aptitude install libdevel-cover-perl + + # From the CPAN with cpanminus + curl -L http://cpanmin.us | perl - --sudo --self-upgrade + cpanm --sudo Devel::Cover + +Then, at the top-level: + + make cover_db_html + +That'll generate a detailed cover report in the "cover_db_html" +directory, which you can then copy to a webserver, or inspect locally +in a browser. + +Smoke testing +------------- + +The Git test suite has support for smoke testing. Smoke testing is +when you submit the results of a test run to a central server for +analysis and aggregation. + +Running a smoke tester is an easy and valuable way of contributing to +Git development, particularly if you have access to an uncommon OS on +obscure hardware. + +After building Git you can generate a smoke report like this in the +"t" directory: + + make clean smoke + +You can also pass arguments via the environment. This should make it +faster: + + GIT_TEST_OPTS='--root=/dev/shm' TEST_JOBS=10 make clean smoke + +The "smoke" target will run the Git test suite with Perl's +"TAP::Harness" module, and package up the results in a .tar.gz archive +with "TAP::Harness::Archive". The former is included with Perl v5.10.1 +or later, but you'll need to install the latter from the CPAN. See the +"Test coverage" section above for how you might do that. + +Once the "smoke" target finishes you'll see a message like this: + + TAP Archive created at <path to git>/t/test-results/git-smoke.tar.gz + +To upload the smoke report you need to have curl(1) installed, then +do: + + make smoke_report + +To upload the report anonymously. Hopefully that'll return something +like "Reported #7 added.". + +If you're going to be uploading reports frequently please request a +user account by E-Mailing gitsmoke@v.nix.is. Once you have a username +and password you'll be able to do: + + SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> make smoke_report + +You can also add an additional comment to attach to the report, and/or +a comma separated list of tags: + + SMOKE_USERNAME=<username> SMOKE_PASSWORD=<password> \ + SMOKE_COMMENT=<comment> SMOKE_TAGS=<tags> \ + make smoke_report + +Once the report is uploaded it'll be made available at +http://smoke.git.nix.is, here's an overview of Recent Smoke Reports +for Git: + + http://smoke.git.nix.is/app/projects/smoke_reports/1 + +The reports will also be mirrored to GitHub every few hours: + + http://github.com/gitsmoke/smoke-reports + +The Smolder SQLite database is also mirrored and made available for +download: + + http://github.com/gitsmoke/smoke-database + +Note that the database includes hashed (with crypt()) user passwords +and E-Mail addresses. Don't use a valuable password for the smoke +service if you have an account, or an E-Mail address you don't want to +be publicly known. The user accounts are just meant to be convenient +labels, they're not meant to be secure. |