Codecov integration

[voronoipotato]

Hey are there any plans to integrate with codecov? Should this be done by Fake? It seems like a great way to evangelize expecto and F# testing in general.

[baronfel]

We'd need to add support for expecto as an OpenCover Strategy, like others do here: https://github.com/OpenCover/opencover/tree/master/main/OpenCover.Extensions/Strategy

[AnthonyLloyd]

No plans but I would be very open to it. I'm not sure what would be required on the Expecto side.

[voronoipotato]

I don't know if this is helpful but...

TrackExpectoTestMethods

using System.Collections.Generic;

namespace OpenCover.Extensions.Strategy
{
    /// <summary>
    /// Track Expecto test methods
    /// </summary>
    public class TrackExpectoTestMethods : TrackedMethodStrategyBase
    {
        private const string ExpectoStrategyName = "ExpectoTest";

        private static readonly IList<string> TrackedAttributeTypeNames = new List<string>
        {
         "Expecto.FTestsAttribute" ,
         "Expecto.TestsAttribute",
         "Expecto.PTestsAttribute" ,
        };

        public TrackExpectoTestMethods() : base(ExpectoStrategyName, TrackedAttributeTypeNames)
        {            
        }
    }
}

https://github.com/OpenCover/opencover/blob/master/main/OpenCover.Extensions/RegisterStrategiesModule.cs

need to add

builder.RegisterType<TrackExpectoTestMethods>().As<ITrackedMethodStrategy>();

https://github.com/OpenCover/opencover/blob/master/main/OpenCover.Test/Extensions/RegisterStrategiesModuleTests.cs

and add

Assert.IsTrue(strategies.Any(s => s is TrackExpectoTestMethods));

and then some version of this?

https://github.com/OpenCover/opencover/blob/master/main/OpenCover.Test/Extensions/Strategy/TrackNUnitTestMethodsTests.cs

[voronoipotato]

I'll be honest I'm relatively new to F#, new to codecov and super duper new to expecto, so while I'm happy to help I also definitely need some sanity checks here.

[baronfel]

@voronoipotato This is a good initial start!

The hard part about extending codevoc to understand expecto is that expecto has 2 modes of operation:

• attribute-based
• code-based

What I mean by this is that expecto allows you to explicitly construct a list of TestCases and run them, as opposed to the reflection Attribute-based detection methods.

So this method would cover the first way, but not the second way.

[voronoipotato]

Cool so by some contrived definition we're halfway done :P

[voronoipotato]

It seems like every test list is always under a test attribute in the samples. Is it possible to use a testlist without a test attribute? If it's not possible to use a testlist without the test attribute I think we might be able to check for the testlists under the TrackExpectoTestMethods constructor. Somehow.

[baronfel]

The samples that use attributes is the runner via 'runTestsInAssembly', which uses reflection. There's also runTests, which takes a text value and runs it. This is used in the second case I mention above.

[voronoipotato]

OpenCover says they only run on windows. :(

[voronoipotato]

I guess there's https://github.com/SteveGilham/altcover which does work on linux, but it's also run by a single superman, which is somewhat concerning.

[haf]

We need anything added to core expecto to work cross platform. So if you have a super-man, then let's use his super-work ;)

May I suggest an alternative? Simpler even.

Instead of open Expecto at the top, we do open Expecto.WithCoverage and that has a 1:1 mapping of the Expecto-module's API, but with callbacks into the coverage tracking framework; that way, there's no need to use reflection and this library doesn't take further steps away being able to run on Node.js with Fable.

[baronfel]

@haf as far as I'm aware everything we've been discussing here is in the context of changes to external tools like openCover or altcover. These tools run your test runner for you after discovering the test methods through various means (like reflection), and these discovery methods are what need to be implemented per-adapter.

So I don't think we'll need to change much in expecto itself, rather just implement logic in openCover/altcover to learn how to discover expecto tests via attribute or by looking for values of type 'TestCase', etc.

[haf]

The problem you're trying to find a decidable solution to is Gödel's Halting Problem, and as such you can't solve it by reflection (decidedly).

[haf]

That said, you could do what VSCode does and scan the IL with Cecil; might be able to pick his code straight from the source. Of course, you won't catch computed tests or async tests, due to the above reason (and the fact that there's no code-evaluator shipped with Cecil)

[AnthonyLloyd]

Can I check what you need from the test functions. They can be parameterised in interesting ways so that may be a factor.

I don't fully know how Codecov works.

Do you need to know the each test function and the code it calls? Then you instrument all that code?

I thinking there could be a way to get the information you need from calling the test exe.

[voronoipotato]

Disclaimer I don't know what I'm doing

Codecov consumes an xml report of what lines of what files were hit by running the testing framework.

OpenCov uses reflection, figures out what tests are run and then tries to figure out what lines of code those tests call and what paths they went down using the PDB.

So basically instead of trying to figure out all the possible paths a test could go, it just observes what paths the tests DID go. This does make your coverage report theoretically nondeterministic with fscheck, but in practice it's fine for the level of fidelity that code coverage reports are for. Code coverage is really a rough way to communicate the existence of tests. 90% coverage doesn't assure any of that coverage is meaningful, but 20% coverage is a good way to communicate that this project is WIP.

[voronoipotato]

https://www.npmjs.com/package/codecov.io

Seems to say they accept lcov, gcov or standardized json data format

[AnthonyLloyd]

Sorry I don't quite understand. Does it instrument the code and run the tests to observe the paths? Or does it work it all out from just reflection?

If it instruments the code then the test exe could list the tests and then they could be run one by one recording the paths.

[voronoipotato]

Opencov as I understand it instruments the code.

[AnthonyLloyd]

Then it may be possible that this can be done using a new Expecto.SDK we are thinking of building.

Something like the following maybe:

1. Expecto.SDK is called to request a list of test assemblies from a root directory.
2. The assemblies are instrumented either in place or not.
3. Expecto.SDK is called to request the list of tests in an assembly.
4. For each test Expecto.SDK is called to run that single test.

[TheAngryByrd]

I started playing with minicover with expecto and they added some more fsharp support here: https://github.com/lucaslorentz/minicover/pull/56

The branch I was playing with: https://github.com/TheAngryByrd/Hopac.Websockets/tree/codecov

To run it call ./codecov.sh

It ends up generating a console report like:

+------------------------------------------+-------+---------------+------------+
| File                                     | Lines | Covered Lines | Percentage |
+------------------------------------------+-------+---------------+------------+
| src/Hopac.Websockets/Hopac.Websockets.fs |  150  |      137      |   91.333%  |
+------------------------------------------+-------+---------------+------------+
| All files                                |  150  |      137      |   91.333%  |
+------------------------------------------+-------+---------------+------------+

And a html report:

I didn't get around to seeing how codecov plays with their xmlreport but it should be supported here: https://github.com/lucaslorentz/minicover/pull/30

[TheAngryByrd]

I've now been able to do something similar with altcover and expecto

#!/bin/bash

pushd ./tests/Hopac.Websockets.Tests

rm -rf bin obj __Instrumented coverage.xml
dotnet build -c Debug

dotnet run --project ~/.nuget/packages/altcover/2.0.330/tools/netcoreapp2.0/AltCover/altcover.core.fsproj -- \
    -i bin/Debug/netcoreapp2.0 --opencover

cp -rf ./__Instrumented/* ./bin/Debug/netcoreapp2.0

dotnet run --project ~/.nuget/packages/altcover/2.0.330/tools/netcoreapp2.0/AltCover/altcover.core.fsproj --configuration Release -- \
    runner -x "dotnet" -w "." -r "bin/Debug/netcoreapp2.0" -- \
    run --no-build --configuration Debug -f netcoreapp2.0


popd

Feeding the https://github.com/danielpalme/ReportGenerator (works with mono) I got a decent report

One thing I've noticed is it slows my tests way down.

Normal:

[21:27:27 INF] EXPECTO! 4 tests run in 00:00:00.6716170 – 4 passed, 0 ignored, 0 failed, 0 errored. ᕙ໒( ˵ ಠ ╭͜ʖ╮ ಠೃ ˵ )७ᕗ <Expecto>

Altcover weaved:

[21:20:33 INF] EXPECTO! 4 tests run in 00:01:58.2490870 – 3 passed, 1 ignored, 0 failed, 0 errored. ᕙ໒( ˵ ಠ ╭͜ʖ╮ ಠೃ ˵ )७ᕗ <Expecto>

But at the very least these other coverage tools do work with Expecto so hopefully this gives someone a place to work from.

[64J0]

Any updates here people?

[TheAngryByrd]

You can use AltCover and YoloDev.Expecto.TestSdk with dotnet test to generate coverage reports. See miniscaffold for a bit deeper look.

[64J0]

Great, thanks @TheAngryByrd

[64J0]

Did anyone tested https://devblogs.microsoft.com/dotnet/whats-new-in-our-code-coverage-tooling/?

---

Update:

I think it's working. This is the report I generated in my project Fubernetes: