Yes, code contracts increase build time. There is work to improve it, but it will take longer with code contracts.
The slowest part is the static check, so you can enable it only for some collections or something like that ... it depends on where the slowdown is causing you problems.