I delete the build directory all the time. This is even recommended when you use svn to ignore, including the assembly directory, since it messed up the assembly.
It just stores files with predefined objects among some other temporary files, so you will not recompile it all from scratch with every build.
.