Testing the statement that something should not compile

Question

Problem

When I work with libraries that support level programming, I often find that I write comments similar to the following (from the example presented by Paul Snively at Strange Loop 2012 ):

// But these invalid sequences don't compile: // isValid(_3 :: _1 :: _5 :: _8 :: _8 :: _2 :: _8 :: _6 :: _5 :: HNil) // isValid(_3 :: _4 :: _5 :: _8 :: _8 :: _2 :: _8 :: _6 :: HNil) 

Or this, from an example in Shapeless :

 /** * If we wanted to confirm that the list uniquely contains `Foo` or any * subtype of `Foo`, we could first use `unifySubtypes` to upcast any * subtypes of `Foo` in the list to `Foo`. * * The following would not compile, for example: */ //stuff.unifySubtypes[Foo].unique[Foo] 

This is a very crude way to point out a fact about the behavior of these methods, and we could assume that we want to make these statements more formal - to test units or regressions, etc.

To give a concrete example of why this might be useful in the context of a Shapeless type library, a few days ago I wrote the following as a quick first attempt at answering this question :

 import shapeless._ implicit class Uniqueable[L <: HList](l: L) { def unique[A](implicit ev: FilterAux[L, A, A :: HNil]) = ev(l).head } 

If the intention is that this will be compiled:

 ('a' :: 'b :: HNil).unique[Char] 

Until it is:

 ('a' :: 'b' :: HNil).unique[Char] 

I was surprised to find that this unique type implementation for HList does not work, because Shapeless would happily find an instance of FilterAux in the latter case. In other words, the following compilation, although you probably expect it to not be:

 implicitly[FilterAux[Char :: Char :: HNil, Char, Char :: HNil]] 

In this case, what I saw was a mistake — or at least something — a mistake — and it has since been fixed .

More generally, we can assume that you want to check the type of invariant that was hidden in my expectations regarding how FilterAux should work with something like unit test - oddly enough, because it might seem like a test type , like this one, with all the recent debates about the relative dignity of types versus tests.

My question

The problem is that I don’t know of any testing framework (for any platform) that allows the programmer to say that something should not be compiled.

One approach I can imagine for the FilterAux case is to use an old trick with an implicit argument with a null value :

 def assertNoInstanceOf[T](implicit instance: T = null) = assert(instance == null) 

So that you can write the following in unit test:

 assertNoInstanceOf[FilterAux[Char :: Char :: HNil, Char, Char :: HNil]] 

The following would be more convenient and expressive:

 assertDoesntCompile(('a' :: 'b' :: HNil).unique[Char]) 

I want this. My question is: does anyone know of any test library or framework that supports anything remotely, like this is perfect for Scala, but I will agree to something.

Answer 1

Not a framework, but Jorge Ortiz ( @JorgeO ) mentioned some utilities that he added to the 2012 Nescala Foursquare Rogue library tests, which has support tests for non-compilation: here you can find examples here . I have long wanted to add something like formless.

More recently, Roland Kuhn ( @rolandkuhn ) added a similar mechanism, this time using Scala 2.10 runtime compilation, before tests for Akka typed channels .

These, of course, are dynamic tests: they do not work (test) if there is something that should not be compiled. Untyped macros can provide a static parameter: i.e. a macro can take an untyped tree, type check it and throw a type error if it succeeds). This may be something that could be experimented with a maneuver-racial formless branch. But not a solution for 2.10.0 or earlier, obviously.

Update

Given the answer to another question, due to Stefan Zeiger ( @StefanZeiger ), surfaced . This is interesting because, like the untyped macro mentioned above, this is compilation time, not verification (verification), but it is also compatible with Scala 2.10.x. Therefore, I consider Roland's approach to be preferable.

Now I have added non-formal options for 2.9.x using the Jorge approach , for 2.10.x using the Stefan approach and for macro-paradise using an untyped macro . Examples of relevant tests can be found here for 2.9.x , here for 2.10.x and here for macro-paradise .

Unexplored macroprocessors are the cleanest, but compatibility with Stefan 2.10.x is a near second.

Answer 2

ScalaTest 2.1.0 has the following syntax for Assertions :

 assertTypeError("val s: String = 1") 

And for Matchers :

 "val s: String = 1" shouldNot compile 

Answer 3

Do you know about partest in the Scala project? For example. CompilerTest has the following document:

 /** For testing compiler internals directly. * Each source code string in "sources" will be compiled, and * the check function will be called with the source code and the * resulting CompilationUnit. The check implementation should * test for what it wants to test and fail (via assert or other * exception) if it is not happy. */ 

He can check if this source can https://github.com/scala/scala/blob/master/test/files/neg/divergent-implicit.scala get https://github.com/scala/scala/blob/ master / test / files / neg / divergent-implicit.check

This is not suitable for your question (since you are not specifying your test cases in terms of statements), but may be an approach and / or give you a start.

Answer 4

Based on the links provided by Miles Sabin , I was able to use the akka version

 import scala.tools.reflect.ToolBox object TestUtils { def eval(code: String, compileOptions: String = "-cp target/classes"): Any = { val tb = mkToolbox(compileOptions) tb.eval(tb.parse(code)) } def mkToolbox(compileOptions: String = ""): ToolBox[_ <: scala.reflect.api.Universe] = { val m = scala.reflect.runtime.currentMirror m.mkToolBox(options = compileOptions) } } 

Then in my tests I used it like this:

 def result = TestUtils.eval( """|import ee.ui.events.Event |import ee.ui.events.ReadOnlyEvent | |val myObj = new { | private val writableEvent = Event[Int] | val event:ReadOnlyEvent[Int] = writableEvent |} | |// will not compile: |myObj.event.fire |""".stripMargin) result must throwA[ToolBoxError].like { case e => e.getMessage must contain("value fire is not a member of ee.ui.events.ReadOnlyEvent[Int]") } 

Answer 5

The compileError macro in μTest does just this:

 compileError("true * false") // CompileError.Type("value * is not a member of Boolean") compileError("(}") // CompileError.Parse("')' expected but '}' found.")