In addition to starting the profiler and checking also on page life expectancy , as well as buffer cache hit ratio See: Use sys.dm_os_performance_counters to get the hit rate of the buffer cache and page life expectancy counters
it can also be (but you have to check) what is happening, is that the data you are looking for has gone out of cache to RAM, now it should get it from disk and it will take longer when you start it again in a second, he will be fast again.
you can check by doing io on statistics
SET STATISTICS IO ON select * ..your query
you should see something like this
Table 'TableNAme'. Scan Number 1, Logical Read 4, Physical Read 2
if you see physical reads above 0, he grabbed it from disk
you can check it (not in production)
deleting data from RAM
DBCC freeproccache DBCC DROPcleanbuffers
now when you run the query twice, you will see something like this, the first run will be from disk, the second from RAM
Table 'TableNAme'. Scan Number 1, Logical Read 4, Physical Read 2
Table 'TableNAme'. Scan Number 1, Logical Read 4, Physical Read 0