Short version: the best way to automate compatibility testing against a large number of third-party programs?
Details:
I am developing a program whose main function interacts with various various software components for music players through the corresponding RPC interfaces. RPC itself usually occurs either through the D-Bus or through a client-specific client library. Since each music player has its own unique RPC interface, my program requires special code to process each.
Testing all this code for me is increasingly becoming a problem. The last count is fifteen (!) Different music players, my program knows how to speak to, and the details of the interface can vary from one version of the player to the next. Manually testing my program for the latest version of each of the players that I am trying to support, as well as several older versions, tedious and error-prone, so I'm looking for a way to automate this as much as possible.
The tests themselves are not a problem; it is only a matter of calling a sequence of functions on the player’s RPC interface and checking the return values and / or asynchronous callbacks for the expected result. No, the problem is that you have the basis for running tests automatically.
Here are the problems I see:
Each player maintains a constant state, usually as dotfiles under a custom home directory. A state is made up of things like library music, playlists, etc. These files should be returned to a known initial state before each test. (Removing the whole is not always, because then the players on the basis of the graphical interface will present the installation wizard the next time they start instead of the usual launch.)
These initial states can be partially dynamic. For example, the construction work of the library will contain full paths to the music files inside it, but the paths to the actual “music” files used for testing will differ from machine to machine and will not be known until runtime.
, , ,
, ,
, .
, , , ,
, ,
.
,
, ,
, .
, (
),
. ,
, -
, -, ,
.
, ,
( ) ?
, , Python,
GNU autotools .