README

NAME

    Test2::Plugin::Cover - Fast and Minimal file coverage info.

DESCRIPTION

    This plugin will collect minimal file coverage info, and will do so
    with minimal performance impact.

    Every time a subroutine is called this tool will do its best to find
    the filename the subroutine was defined in, and add it to a list. Also,
    anytime you attempt to open a file with open() or sysopen() the file
    will be added to the list. This list will be attached to a test2 event
    just before the test exits. In most formatters the event will only show
    up as a comment on STDOUT  # This test covered N source files. .
    However tools such as Test2::Harness::UI can make full use of the
    coverage information contained in the event.

INTENDED USE CASE

    This tool is not intended to record comprehensive coverage information,
    if you want that use Devel::Cover.

    This tool is intended to obtain and maintain lists of files that were
    opened, or which define subs which were executed by any given test.
    This information is useful if you want to determine what test files to
    run after any given code change.

    The collected coverage data is contained in test2 events, if you use
    Test2::Harness aka yath then this data can be logged and consumed by
    other tools such as Test2::Harness::UI.

PERFORMANCE

    Unlike tools that need to record comprehensive coverage (Devel::Cover),
    This module is only concerned about what files you open, or defined
    subs executed directly or indirectly by a given test file. As a result
    this module can get away with a tiny bit of XS code that only fires
    when a subroutine is called. Most coverage tools fire off XS for every
    statement.

LIMITATIONS

    This tool uses XS to inject a little bit of C code that runs every time
    a subroutine is called, or every time open() or sysopen() is called.
    This C code obtains the next op that will be run and tries to pull the
    filename from it. eval, XS, Moose, and other magic can sometimes mask
    the filename, this module only makes a minimal attempt to find the
    filename in these cases.

    Originally this module only collected the filenames touched by a test.
    Now in addition to that data it can give you separate lists of files
    where subs were called, and files that were touched via open().
    Additionally the sub list includes the info about what subs were
    called. In all of these cases it is also possible to know what sections
    of your test called the subs or opened the files.

 THINGS THAT WILL NOT SHOW UP

    goto

      goto &sub does not enter the sub the normal way, the target sub (and
      its file, if different) will not be recorded. The sub doing the goto
      is recorded normally.

    constants

      Constants created with use constant, and any other subs inlined at
      compile time, never trigger a runtime sub call, so the file defining
      them will not be recorded unless something else in it is called.

    XS subs

      XS subs have no perl source file. Calls to them are recorded against
      the file of the next perl statement that executes, which is usually
      the caller's file.

    threads

      Coverage data is collected per-thread and is not merged. Data
      collected inside spawned ithreads is lost unless you merge it
      yourself.

    exotic open() forms

      Only 2 and 3 argument open() calls are recorded. The list form for
      piping to programs (open($fh, '-|', $prog, @args)) and 1-arg open()
      are ignored.

 REAL EXAMPLES

    The following data was gathered using prove to run the full Moose test
    suite:

        # Prove on its own
        Files=478, Tests=17326, 64 wallclock secs ( 1.62 usr  0.46 sys + 57.27 cusr  4.92 csys = 64.27 CPU)
    
        # Prove with Test2::Plugin::Cover (no coverage event)
        Files=478, Tests=17326, 67 wallclock secs ( 1.61 usr  0.46 sys + 60.98 cusr  5.31 csys = 68.36 CPU)
    
        # Prove with Devel::Cover
        Files=478, Tests=17324, 963 wallclock secs ( 2.39 usr  0.58 sys + 929.12 cusr 31.98 csys = 964.07 CPU)

    no coverage event - No report was generated. This was done to only
    measure the effect of the XS that adds the data collection overhead,
    and not the cost of the perl code that generates the report event at
    the end of every test.

    The Moose test suite was also run using Test2::Harness aka yath

        # Without Test2::Plugin::Cover
        Wall Time: 62.51 seconds CPU Time: 69.13 seconds (usr: 1.84s | sys: 0.08s | cusr: 60.77s | csys: 6.44s)
    
        # With Test2::Plugin::Cover (no coverage event)
        Wall Time: 75.46 seconds CPU Time: 82.00 seconds (usr: 1.96s | sys: 0.05s | cusr: 72.64s | csys: 7.35s)

    As you can see, there is a performance hit, but it is fairly small,
    specially compared to Devel::Cover. This is not to say anything bad
    about Devel::Cover which is amazing, but a bad choice for the use case
    Test2::Plugin::Cover was written to address.

SYNOPSIS

 INLINE

        use Test2::Plugin::Cover;
    
        ...
    
        # Arrayref of files covered so far
        my $covered_files = Test2::Plugin::Cover->files;

 COMMAND LINE

    You can tell prove to use the module this way:

        HARNESS_PERL_SWITCHES=-MTest2::Plugin::Cover prove ...

    For yath:

        yath test --cover-files ...

 SUPPRESS REPORT

    You can suppress the final report (only collect data, do not send the
    Test2 event)

    CLI:

        HARNESS_PERL_SWITCHES=-MTest2::Plugin::Cover=no_event,1 prove ...

    INLINE:

        use Test2::Plugin::Cover no_event => 1;

 EXCLUDE DIRECTORIES

    A directory under the root, such as a dependency tree installed inside
    the workspace, can be kept out of the coverage data entirely:

    CLI:

        HARNESS_PERL_SWITCHES=-MTest2::Plugin::Cover=exclude,deps prove ...

    INLINE:

        use Test2::Plugin::Cover exclude => 'deps';
    
        # Or several at once
        use Test2::Plugin::Cover exclude => ['deps', 'vendor'];

    An excluded path and everything under it, at any depth, is dropped
    before the coverage event is sent, so excluded files never reach tools
    consuming that event. Paths are compared component by component, so
    excluding lib does not exclude a sibling library. A relative path is
    resolved against the coverage root, so it keeps meaning the same tree
    in a process that has changed directory.

    Wildcards are not supported, an exclusion is a literal path. Excluding
    a whole tree needs only the top of it.

KNOWING WHAT CALLED WHAT

    If you use a system like Test::Class, Test::Class::Moose, or
    Test2::Tools::Spec then you divide your tests into subtests (or
    similar). In these cases it would be nice to track what subtest (or
    equivalent) touched what files.

    There are 3 methods related to this, set_from(), get_from(), and
    clear_from() which you can use to manage this meta-data:

        subtest foo => sub {
            # Note, this is a simple string, but the 'from' data can also be a data
            # structure.
            Test2::Plugin::Cover->set_from("foo");
    
            # subroutine() from Some.pm will be recorded as having been called by 'foo'.
            Some::subroutine();
    
            Test2::Plugin::Cover->clear_from();
        };

    Doing this manually for all blocks is not ideal, ideally you would hook
    your tool, such as Test::Class to call set_from() and clear_from() for
    you. Adding such a hook is left as an exercise to the reader, and if
    you make one for a popular tool please upload it to cpan and add a
    ticket or send an email for me to link to it here.

    Once you have these hooks in place the data will not only show files
    and subs that were called, but what called them.

    Please see the set_from() documentation for details on values.

CLASS METHODS

    $class->touch_source_file($file)

    $class->touch_source_file($file, $sub)

    $class->touch_source_file($file, \@subs)

    $class->touch_source_file($file, $subs, $from)

      This can be used to manually add coverage data. The first argument is
      the source file to be "touched" by coverage. The second argument is
      optional, and may be either a subroutine name, or an arrayref of
      subroutine names. The third argument is optional and can be used to
      override the default "from" value, which is normally determined for
      you automatically.

      If no subroutines are specified it will default to using '*', which
      means the entire file is considered to be touched.

    $class->touch_data_file($file)

    $class->touch_data_file($file, $from)

      This can be used to manually add coverage data. The first argument is
      the file to be "touched" by coverage data. Optionally you can
      override the 'from' value which is normally determined automatically.

      This is the same as calling $class->touch_source_file($file, '<>').

      Both touch methods are no-ops while coverage is disabled, see
      disable().

    $class->enable()

    $class->disable()

    $bool = $class->enabled()

      Toggle or check enabled status. When disabled no coverage is
      recorded.

    $class->reload()

      Reset filter if $0 or __FILE__ have changed. This is advanced usage,
      you will probably never need this.

    $val = $class->get_from()

      Get the current 'from' value. The default is '*' when nothing has set
      a from value.

    $class->set_from($val)

      Set a 'from' value. This can be anything, a string, a hashref, etc.
      Be advised though that it will usually be serialized to JSON, so make
      sure anything you put in it will be serializable as json.

      If the value is, or contains, a CODE or GLOB reference it cannot be
      serialized into the final report, so a warning will be issued and the
      value will be ignored (the previous 'from' value stays in effect).
      This is not fatal because enabling coverage should never introduce
      new exceptions into the code being observed.

    $class->clear_from()

      Resets the clear value to '*'

    $bool = $class->was_from_modified()

      This will return true if anything has called set_from() or
      set_from_manager. This can be reset back to false using reset_from(),
      which also clears the 'from' and 'from_manager' values.

    $class->set_from_manager($module)

      This should be set to a module that implements the following method:

          sub test_parameters {
              my $class = shift;
              my ($test_file, \@from_values) = @_;
      
              ...
      
              return {
                  # If true - run the test
                  # If false - skip the test
                  # If not present or undef - run the test
                  run => $bool,
      
                  # The following are optional
                  argv  => [ ... ],
                  env   => { ... },
                  stdin => "...",
              };
      
              # OR
              # If true - run the test
              # If false - skip the test
              # If undef or empty list - run the test
              return $bool;
          }

      This will be used by Test2::Harness to determine what data needs to
      be passed to a test given a set of 'from' values to instruct the test
      to run the necessary parts/subtests/groups/methods/etc.

      The 'argv' data will be prepended before any other arguments provided
      to the test.

      The 'env' hashref will be merged with any other env vars needed, with
      these taking priority.

      The 'stdin' string will be used as STDIN for the test.

    $arrayref = $class->files()

    $arrayref = $class->files(root => $path)

    $arrayref = $class->files(exclude => $path_or_arrayref)

      This will return an arrayref of all files touched so far.

      The list of files will be sorted alphabetically, and duplicates will
      be removed.

      If a root path is provided it may be a Path::Tiny instance or a plain
      string. This path will be used to filter out any files not under the
      root directory.

      If an exclude path is provided, that path and everything under it is
      left out of the results. It may be a single path or an arrayref of
      them, each a Path::Tiny instance or a plain string.

      The running test file ($0) and this plugin's own file are always
      excluded from the results.

    $hashref = $class->data()

    $hashref = $class->data(root => $path)

    $hashref = $class->data(exclude => $path_or_arrayref)

      This returns the processed coverage data that goes into the report
      event:

          {
              # Files where subs were called
              'lib/Foo.pm' => {
                  some_sub => [ list of 'from' values that called it ],
      
                  # Called BEGIN/END/etc blocks, or subs whose name could not be
                  # determined, fall under '*'.
                  '*' => [ ... ],
              },
      
              # Files opened with open()/sysopen(), or touched as data files
              'data.json' => { '<>' => [ ... ] },
          }

      Duplicate 'from' values are removed (compared by content, not
      reference), and each list is sorted deterministically. The root and
      exclude parameters behave as they do in files().

    $event = $class->report(%options)

      This will send a Test2 event containing coverage information. It will
      also return the event.

      The event's coverage facet carries the processed file data from
      data() under files, and under root the canonical absolute directory a
      relative file path is resolved against, so a consumer in another
      process can resolve them without guessing.

      Options:

      root => Path::Tiny->new("...")

	Normally this is set to the current directory at module load-time.
	This is used to filter out any source files that do not live under
	the current directory. This may be a Path::Tiny instance or a plain
	string.

      exclude => $path_or_arrayref

	Paths to leave out of the report entirely, along with everything
	under them. May be a single path or an arrayref of them, each a
	Path::Tiny instance or a plain string. Relative paths are resolved
	against the coverage root when one is supplied, and against the
	current directory otherwise. Wildcards are not supported.

	When passed at import time this option may be given more than once
	instead of using an arrayref, which is how it survives the -M
	command line form.

      verbose => $BOOL

	If this is set to true then the comment stating how many source
	files were touched will be printed as a diagnostics message instead
	so that it shows up without a verbose harness.

      ctx => DO NOT USE

	This is used ONLY when the Test2::API is doing its final
	book-keeping. Most users will never want to use this.

    $class->reset_coverage()

      This will completely clear all coverage data so far.

    $class->reset_from()

      This will clear the 'from' value, as well as reset the
      'was_from_modified' state to false.

    $class->full_reset()

      Calls both reset_coverage() and reset_from().

    $file_or_undef = $class->filter($file)

    $file_or_undef = $class->filter($file, root => Path::Tiny->new('...'))

    $file_or_undef = $class->filter($file, exclude => ['...'])

      This method is used as a callback when getting the final list of
      covered source files. The default implementation removes any files
      that are not under the current directory which lets you focus on
      files in the distribution you are testing. You may return a modified
      filename if you wish to normalize it here, the default implementation
      will turn it into a relative path.

      A returned path must be relative to the root, or absolute. The
      coverage event names that root so that consumers can resolve relative
      paths, and a relative path that is not actually under it names the
      wrong file.

      If you provide a custom root parameter, it may be a Path::Tiny
      instance or a plain string.

      Exclusions are applied here, so a subclass that replaces this method
      is responsible for honoring the exclude parameter if it needs
      exclusions to keep working.

      A custom filter callback should look something like this:

          sub {
              my $class = shift;
              my ($file, %params) = @_;
      
              # clean_filename() does not exist, it is just an example
              $file = clean_filename($file, %params);
      
              # should_show() does not exist, it is just an example
              return $file if should_show(%params);
      
              # Return undef or an empty list if you do NOT want to show the file.
              return;
          }

      Please take a look at the source to see what and how filter() is
      implemented if you want all the details on how it works.

    $file_or_undef = $class->extract($file)

    $file_or_undef = $class->extract($file, %params)

      This method is responsible for extracting a sensible filename from
      whatever the XS found. Some magic such as eval or Moose can set the
      filename to strings like '(eval 123)' or 'foo bar (defined at FILE
      line LINE)' or even nonsensical strings, or text with no filenames.

      If a sensible file name can be extracted it will be returned,
      otherwise undef (or an empty list) is returned.

      The default implementation does not use any parameters, but they are
      passed in for custom implementations to use.

      A custom extract callback should look something like this:

          sub {
              my $class = shift;
              my ($file, %params) = @_;
      
              # It is a valid file
              return $file if -e $file;
      
              # Do not use this, just an example
              return $1 if $file =~ m/($VALID_FILE_REGEX)/;
      
              # Cannot find a file here
              return;
          }

TRACING OPENS

    For debugging you can ask the plugin to record where every open() and
    sysopen() call happened:

        $Test2::Plugin::Cover::TRACE_OPENS = 1;

    Every recorded open will push an arrayref onto
    @Test2::Plugin::Cover::OPENS:

        [$filename, $file, $line, $package]

    Where $filename is what was being opened, and $file, $line and $package
    describe the code doing the open. This is a debugging aid only, the
    data is not included in coverage events.

SEE ALSO

    Devel::Cover is by far the best and most complete coverage tool for
    perl. If you need comprehensive coverage use Devel::Cover.
    Test2::Plugin::Cover is only better for a limited use case.

SOURCE

    The source code repository for Test2-Plugin-Cover can be found at
    https://github.com/Test-More/Test2-Plugin-Cover.

MAINTAINERS

    Chad Granum <exodist@cpan.org>

AUTHORS

    Chad Granum <exodist@cpan.org>

COPYRIGHT

    Copyright 2020 Chad Granum <exodist@cpan.org>.

    This program is free software; you can redistribute it and/or modify it
    under the same terms as Perl itself.

    See http://dev.perl.org/licenses/