Update the plugin to allow a configurable start

[costin-zaharia]

Adds nunitCommandModifier config which if set, can overwrite
the default command arguments in order to allow running of
external tools like dotMemoryUnit.

[panteamihai]

LGTM

[dfev77]

should add some tests to cover the new functionality

[costin-zaharia]

@dfev77 I've added unit tests to cover the new functionality

[panteamihai]

+1

[gluck]

Seems like you took a different approach for integrating dotMemory than what was done for opencover, is it on purpose ?
In particular, not having a dotMemory plugin means that the executable will have to be installed/available on the machine running the build ?
Also will it still integrate nicely with opencover, or will users have to choose between coverage and memory tests ?

[costin-zaharia]

Seems like you took a different approach for integrating dotMemory than what was done for opencover, is it on purpose ?

ignorance not purpose :) I will take a look on opencover to see if I can apply the same approach

In particular, not having a dotMemory plugin means that the executable will have to be installed/available on the machine running the build ?

not needed since the dotMemoryUnit.exe tool is part of the referenced nuget package

Also will it still integrate nicely with opencover, or will users have to choose between coverage and memory tests ?

I will check and come back with an answer

[costin-zaharia]

Seems like you took a different approach for integrating dotMemory than what was done for opencover, is it on purpose ?

I've took a quick look on the opencover nunit plugin and that approach seems to be a lot more extensible for future improvements but, as far as I can tell, for the moment we don't need that extra flexibility since there is no other customisation needed than running the tests on dotMemory environment.
In this specific case the tool is already downloaded by the nuget restore.

An usage example: https://blog.jetbrains.com/dotnet/2015/09/15/memory-testing-on-a-ci-server-dotmemory-unit-standalone-launcher/

dotMemoryUnit.exe -targetExecutable="C:\NUnit 2.6.4\bin\nunit-console.exe" -returnTargetExitCode --"E:\MyProject\bin\Release\MainTests.dll"

From another point of view the nunitCommandModifier is generic enough to be used in other contexts by providing a way for customising the arguments passed to the runner. (Maybe a fix for #30?)

Also will it still integrate nicely with opencover, or will users have to choose between coverage and memory tests ?

As far as I can tell, opencover is building it's own command ignoring the nunit one so they will integrate nice but there is a downside: since both dotMemory and opencover need the tests to be run in their own environment, the users will have to run the tests twice if both coverage and memory testing is needed.

[gluck]

Irt #30 it could be used, though I don't think that approach is good, for the same reason(s) I mentioned on the ticket.
How does dotMemory assertions behave when ran through plain nunit (or opencover) ? won't they fail the tests ? Shouldn't we have something like opencover run dotmemory run nunit ?

[costin-zaharia]

How does dotMemory assertions behave when ran through plain nunit (or opencover) ? won't they fail the tests?

The behavior of the memory tests can be controlled using an attribute: [DotMemoryUnit(FailIfRunWithoutSupport = false)]

Shouldn't we have something like opencover run dotmemory run nunit ?

I will try to see if it works fine.

[costin-zaharia]

Shouldn't we have something like opencover run dotmemory run nunit ?

I was not able to run them like you sugested: opencover -> dotmemory -> nunit due to what I assume is an opencover profiler problem:

Committing...
No results, this could be for a number of reasons. The most common reasons are:
1) missing PDBs for the assemblies that match the filter please review the
output file and refer to the Usage guide (Usage.rtf) about filters.
2) the profiler may not be registered correctly, please refer to the Usage
guide and the -register switch.

but I was able to run them the other way around: dotmemoryunit -> opencover -> nunit