What testing unit should I use for Qt?

Question

I'm just starting a new project that needs a cross-platform graphical interface, and we chose Qt as a graphical interface.

We also need a unit testing infrastructure. Until a year ago, we used the built-in platform for testing modules for C ++ projects, but now we are moving on to using Google Test for new projects.

Does anyone have experience using Google Test for Qt applications? Is QtTest / QTestLib a better alternative?

I'm still not sure how much we want to use Qt in non-GUI parts of the project — we probably would rather just use STL / Boost in the main code with a small interface to the Qt-based GUI.

EDIT: Many seem to be leaning towards QtTest. Is there anyone who has experience integrating with a continuous integration server? In addition, it seemed to me that the need to process a separate application for each new test case would cause a lot of friction. Is there a good way to solve this? Does Qt Creator have a good way to handle such test cases, or do you need a project for a test case?

Answer 1

I do not know that QTestLib is "better" than one structure for another in such general terms. There is one thing that it does well, and it is a good way to test Qt-based applications.

You can integrate QTest into a new setup based on Google Test. I haven't tried it, but based on how QTestLib is archived, it seems like it won't be too complicated.

Tests written with pure QTestLib have the -xml option, which you can use with some XSLT transformations to convert to the required format for the continuous integration server. However, a lot depends on which CI server you are working with. I would suggest that this also applies to GTest.

One test application for a test case never caused much friction for me, but it depends on the availability of an assembly system that could do a decent job of managing the building and performing test cases.

I don’t know anything in Qt Creator, which will require a separate project for each test case, but it can change since the last time I looked at Qt Creator.

I also suggest sticking to QtCore and staying away from STL. Using QtCore throughout will deal with GUI bits that require Qt data types. In this case, you don’t have to worry about switching from one data type to another.

Answer 2

You do not need to create separate test applications. Just use qExec in an independent main () function like this:

int main(int argc, char *argv[]) { TestClass1 test1; QTest::qExec(&test1, argc, argv); TestClass2 test2; QTest::qExec(&test2, argc, argv); // ... return 0; } 

This will perform all testing methods in each class in one batch.

Your testclass.h files look like this:

 class TestClass1 : public QObject { Q_OBJECT private slots: void testMethod1(); // ... } 

Unfortunately, this setting is not well described in the Qt documentation, although it seems to be very useful for many people.

Answer 3

To add an answer to Joe.

Here is a small heading that I use (testrunner.h), which contains a utility class that generates an event loop (which, for example, is necessary to test connections and databases connected to the signal slot) and "working" classes compatible with QTest:

 #ifndef TESTRUNNER_H #define TESTRUNNER_H #include <QList> #include <QTimer> #include <QCoreApplication> #include <QtTest> class TestRunner: public QObject { Q_OBJECT public: TestRunner() : m_overallResult(0) {} void addTest(QObject * test) { test->setParent(this); m_tests.append(test); } bool runTests() { int argc =0; char * argv[] = {0}; QCoreApplication app(argc, argv); QTimer::singleShot(0, this, SLOT(run()) ); app.exec(); return m_overallResult == 0; } private slots: void run() { doRunTests(); QCoreApplication::instance()->quit(); } private: void doRunTests() { foreach (QObject * test, m_tests) { m_overallResult|= QTest::qExec(test); } } QList<QObject *> m_tests; int m_overallResult; }; #endif // TESTRUNNER_H 

Use it as follows:

 #include "testrunner.h" #include "..." // header for your QTest compatible class here #include <QDebug> int main() { TestRunner testRunner; testRunner.addTest(new ...()); //your QTest compatible class here qDebug() << "Overall result: " << (testRunner.runTests()?"PASS":"FAIL"); return 0; } 

Answer 4

I started using QtTest for my application and very quickly began to run into limitations. Two main problems:

1) My tests run very quickly - fast enough that the overhead of downloading the executable file, setting up the Q (Core) application (if necessary), etc. often outshine the running time of the tests themselves! Linking each executable takes a lot of time.

The overhead simply increased as more and more classes were added, and soon it became a problem - one of the goals of unit tests was to have a safety net that worked so fast that it wasn’t a burden, and it quickly didn’t. The solution is to combine several test suites into one executable file, and while (as shown above) this basically does not work, is not supported, and has important limitations.

2) Lack of hardware support is a deal breaker for me.

So, after a while I switched to Google Test - it is a much more functional and complex unit testing system (especially when used with Google Mock) and solves 1) and 2), and in addition, you can still easily use the convenient QTestLib functions such as QSignalSpy and GUI simulation, etc. It was a little painful to switch, but, fortunately, the project did not go too far, and many of these changes can be automated.

Personally, I will not use QtTest over Google Test for future projects - unless it offers the real advantages that I see, and has important disadvantages.

Answer 5

Why not use the unit testing framework included in Qt? Example: QtTestLib Tutorial .

Answer 6

QtTest is mainly useful for testing parts requiring Qt loop dispatch. It is designed in such a way that a separate executable file is required for each test case, so it should not conflict with the existing test environment used for the rest of the application.

(By the way, I highly recommend using QtCore even for third-party applications other than the GUI, which are much more convenient to work with.)

Answer 7

To extend the solution to mlvljr and Joe, we can even support the full QtTest parameters by one test class and still run everything in batch writing:

 usage: help: "TestSuite.exe -help" run all test classes (with logging): "TestSuite.exe" print all test classes: "TestSuite.exe -classes" run one test class with QtTest parameters: "TestSuite.exe testClass [options] [testfunctions[:testdata]]... 

Headline

 #ifndef TESTRUNNER_H #define TESTRUNNER_H #include <QList> #include <QTimer> #include <QCoreApplication> #include <QtTest> #include <QStringBuilder> /* Taken from https://stackoverflow.com/questions/1524390/what-unit-testing-framework-should-i-use-for-qt BEWARE: there are some concerns doing so, see https://bugreports.qt.io/browse/QTBUG-23067 */ class TestRunner : public QObject { Q_OBJECT public: TestRunner() : m_overallResult(0) { QDir dir; if (!dir.exists(mTestLogFolder)) { if (!dir.mkdir(mTestLogFolder)) qFatal("Cannot create folder %s", mTestLogFolder); } } void addTest(QObject * test) { test->setParent(this); m_tests.append(test); } bool runTests(int argc, char * argv[]) { QCoreApplication app(argc, argv); QTimer::singleShot(0, this, SLOT(run())); app.exec(); return m_overallResult == 0; } private slots: void run() { doRunTests(); QCoreApplication::instance()->quit(); } private: void doRunTests() { // BEWARE: we assume either no command line parameters or evaluate first parameter ourselves // usage: // help: "TestSuite.exe -help" // run all test classes (with logging): "TestSuite.exe" // print all test classes: "TestSuite.exe -classes" // run one test class with QtTest parameters: "TestSuite.exe testClass [options] [testfunctions[:testdata]]... if (QCoreApplication::arguments().size() > 1 && QCoreApplication::arguments()[1] == "-help") { qDebug() << "Usage:"; qDebug().noquote() << "run all test classes (with logging):\t\t" << qAppName(); qDebug().noquote() << "print all test classes:\t\t\t\t" << qAppName() << "-classes"; qDebug().noquote() << "run one test class with QtTest parameters:\t" << qAppName() << "testClass [options][testfunctions[:testdata]]..."; qDebug().noquote() << "get more help for running one test class:\t" << qAppName() << "testClass -help"; exit(0); } foreach(QObject * test, m_tests) { QStringList arguments; QString testName = test->metaObject()->className(); if (QCoreApplication::arguments().size() > 1) { if (QCoreApplication::arguments()[1] == "-classes") { // only print test classes qDebug().noquote() << testName; continue; } else if (QCoreApplication::arguments()[1] != testName) { continue; } else { arguments = QCoreApplication::arguments(); arguments.removeAt(1); } } else { arguments.append(QCoreApplication::arguments()[0]); // log to console arguments.append("-o"); arguments.append("-,txt"); // log to file as TXT arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".log,txt"); // log to file as XML arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".xml,xunitxml"); } m_overallResult |= QTest::qExec(test, arguments); } } QList<QObject *> m_tests; int m_overallResult; const QString mTestLogFolder = "testLogs"; }; #endif // TESTRUNNER_H 

own code

 #include "testrunner.h" #include "test1" ... #include <QDebug> int main(int argc, char * argv[]) { TestRunner testRunner; //your QTest compatible class here testRunner.addTest(new Test1); testRunner.addTest(new Test2); ... bool pass = testRunner.runTests(argc, argv); qDebug() << "Overall result: " << (pass ? "PASS" : "FAIL"); return pass?0:1; } 

Answer 8

If you use Qt, I would recommend using QtTest, because it has the ability to test the user interface and is easy to use.

If you use QtCore, you can probably do without STL. I often find the Qt classes easier to use than the STL counterparts.

Answer 9

I tested our libraries with gtest and QSignalSpy . Use QSignalSpy to detect signals. You can directly call slots (for example, regular methods).

Answer 10

I just played with it. The main advantage of using Google Test for QtTest for us is that we do all our user interface development in Visual Studio. If you use Visual Studio 2012 and install the Google Test Adapter , you can force VS to recognize the tests and include them in your test explorer. This is great for developers to use when writing code, and since Google Test is portable, we can also add tests to the end of our Linux build.

In the future, I hope someone will add C ++ support to one of the parallel testing tools that C # has, such as NCrunch , Giles, and ContinuousTests .

Of course, you may find that someone is writing another adapter for VS2012 that adds QtTest support to the test adapter, in which case this advantage goes away! If anyone is interested in this, there is a good blog entry Creating a new Visual Studio unit test adapter .

Answer 11

To support the Visual Studio test adapter adapter tool with a QtTest card, this is a Visual Studio extension: https://visualstudiogallery.msdn.microsoft.com/cc1fcd27-4e58-4663-951f-fb02d9ff3653