HTF does not test details created by TH

Question

I want to do some similar tests for different types in my library.

To simplify the situation, suppose I have several types of vectors that implement the Num class, and I want to generate the same check for the QuickCheck property prop_absNorm xy = abs x + abs y >= abs (x+y) , which will work on all types in library.

I generate such properties using TH:

 $(writeTests (\t -> [d| prop_absNorm :: $(t) -> $(t) -> Bool prop_absNorm xy = abs x + abs y >= abs (x+y) |]) ) 

My test generation function has the following signature:

 writeTests :: (TypeQ -> Q [Dec]) -> Q [Dec] 

This function searches for all instances of my VectorMath (n::Nat) t vector class (and, at the same time, Num instances) through reify ''VectorMath and accordingly generates all the support functions. -ddump-splices shows something like this:

 prop_absNormIntX4 :: Vector 4 Int -> Vector 4 Int -> Bool prop_absNormIntX4 xy = abs x + abs y >= abs (x+y) prop_absNormCIntX4 :: Vector 4 CInt -> Vector 4 CInt -> Bool prop_absNormCIntX4 xy = abs x + abs y >= abs (x+y) ... prop_absNormFloatX4 :: Vector 4 Float -> Vector 4 Float -> Bool prop_absNormFloatX4 xy = abs x + abs y >= abs (x+y) prop_absNormFloatX3 :: Vector 3 Float -> Vector 3 Float -> Bool prop_absNormFloatX3 xy = abs x + abs y >= abs (x+y) 

The problem is that all properties written by hand are checked, but not generated.

Note 1: I created and did not generate properties in one file (i.e. the expression TH $(..) is in the same file as the other details).

Note 2: the list of types for creating prop functions is a variable - I want to add other instances of VectorMath later, so they are automatically added to the test list.

I believe the problem is that the HTF (which supposedly uses TH too) is parsing the source file, not the one with the generated code, but I cannot understand why this is happening.

So my question is: how to solve this problem? If it is impossible to use TH-generated details, is it possible to run QuickCheck tests on different types (i.e. Replaces them with prop_absNorm :: Vector 4 a -> Vector 4 a -> Bool )?

Another alternative would be to use TH further to add test entries manually in htf_Main, but I have not figured out how to do this yet; and it doesn't seem like a good clean solution.

Answer 1

Ok, I managed to solve this problem. The idea is to use TH to aggregate tests and paste them into htfMain . In addition to what I have in the question, this includes the following steps:

• Converting all validated properties to IO actions that run QuickCheck tests;
• The totality of all tests in TestSuite ;
• Group all test packages in one list and put it in htfMain .

To use step 1, I had to use a semi-internal HTF function called qcAssertion :: (QCAssertion t) => t -> Assertion . This feature is available but not recommended for external use; it makes it easy to run QuickCheck tests by integrating them into a report.

To go to step 2, I use two functions from HTF: makeTestSuite and makeQuickCheckTest . I also use the location function from TH to specify the file name and the location string where splicing with the test pattern is inserted (for more convenient test logs).

Step 3 is complicated: for this we need to find all the generated test packages. The problem is that TH does not allow viewing all functions (including generated ones) in the module. To overcome this, I added the following type class:

 class MultitypeTestSuite name where multitypeTestSuite :: name -> TestSuite 

So my writeTests function creates a new data type data MTS[prop_name] and an instance of MultitypeTestSuite for this data type. This allows me to use another splice function in htfMain later, which will generate a list of test suites from instances of this class using reify :

 aggregateTests :: ExpQ aggregateTests = do ClassI _ instances <- reify ''MultitypeTestSuite liftM ListE . forM instances $ \... -> [e| multitypeTestSuite $(...) |] 

In the end, including all generated tests along with manually written tests, it looks pretty simple:

 main :: IO () main = htfMain $ htf_importedTests ++ $(aggregateTests) 

So, by setting up the $(writeTests) function, I can now generate and test properties that differ in the type of arguments - for all types available in the area with the same type. Test results and logs are included in the same way as the original tests.

That the problem is completely resolved.

Answer 2

If you know in advance what the names of generated property tests are, then you can always manually define stubs so that HTF can see them, for example:

 $(generate prop test for Int) $(generate prop test for CInt) prop_p1 = prop_absNormInt prop_p2 = prop_absNormCInt 

HTF will see the tests as prop_p1 and prop_p2 . You do not need to put type signatures on these stubs.

Another idea is to create your own pre-processor to add these stubs (and give them better names). Your source preprocessor will automatically call htfpp to complete the preprocessing.

If you show me how your TH is called, I can show you how to write a preprocessor.

Update:

Given your comment, I would look at the following:

• Write a program to generate a test module source.
• Include this program and the output that it generates in your cabal project.
• Let users know if they want to update the test module.

So - test cases remain fixed until the program is launched to regenerate the test module.

Having a static test module has the advantage that you can say for sure what is being tested.

Having a program to recreate the test module gives you the ability to easily update it when new Num instances are available.

Answer 3

HTF does not use TemplateHaskell to collect tests, this will significantly slow down compilation time. Instead, HTF uses a special preprocessor called htfpp . htfpp runs before the compiler (and therefore, before the TemplateHaskell splices expand). This means that you cannot use automatic test detection with htfpp when creating tests using TemplateHaskell.

My suggestion: when you use TemplateHaskell anyway, just use TemplateHaskell to collect your generated test cases. This functionality is not built into HTF, but it is not difficult to implement such a function. Here he is:

 -- file TH.hs {-# LANGUAGE TemplateHaskell #-} module TH ( genTestSuiteFromQcProps ) where import Language.Haskell.TH import Test.Framework import Test.Framework.Location genTestSuiteFromQcProps :: String -> [Name] -> Q Exp genTestSuiteFromQcProps suiteName names = [| makeTestSuite $(stringE suiteName) $(listE genTests) |] where genTests :: [ExpQ] genTests = map genTest names genTest :: Name -> Q Exp genTest name = [| makeQuickCheckTest $(stringE (show name)) unknownLocation (qcAssertion $(varE name)) |] 

The genTestSuiteFromQcProps function takes the name of the generated test suite and a list of names, referencing your QC properties. genTestSuiteFromQcProps returns an expression of type TestSuite . TestSuite is one of the types used by HTF to organize tests. (The htfpp preprocessor uses the TestSuite type in its output.)

This is how you use genTestSuiteFromQcProps :

 -- file Main.hs {-# OPTIONS_GHC -F -pgmF htfpp #-} {-# LANGUAGE TemplateHaskell #-} module Main where import TH import Test.Framework import { -@ HTF_TESTS @-} OtherTests prop_additionCommutative :: Int -> Int -> Bool prop_additionCommutative xy = (x + y) == (y + x) prop_reverseReverseIdentity :: [Int] -> Bool prop_reverseReverseIdentity l = l == reverse (reverse l) myTestSuite :: TestSuite myTestSuite = $(genTestSuiteFromQcProps "MyTestSuite" ['prop_additionCommutative ,'prop_reverseReverseIdentity]) main :: IO () main = htfMain (myTestSuite : htf_importedTests) 

In your case, you pass genTestSuiteFromQcProps names of the QC properties that you created using TemplateHaskell.

This example also shows that you can combine test cases generated using the TemplateHaskell function with test cases built using htfpp . For completeness, here are the contents of OtherTests :

 {-# OPTIONS_GHC -F -pgmF htfpp #-} module OtherTests ( htf_thisModulesTests) where import Test.Framework test_someOtherTest :: IO () test_someOtherTest = assertEqual 1 1