First of all, the continuation will be executed even if a regular exception has occurred. However, it is less likely than a regular finally block in the case of exceptional conditions, such as an OutOfMemoryException.
Now I would not try to rid the web client. Remember that recycling is an optimization, since in any case local resources will be installed by the finalizer. The only reason we get rid of is because the finalizer is expensive because it launches the second pass of the GC.
But to complete your optimization, the system may have to create new threads. In addition, you can extend the life of your web client a lot if the threadpool is full of long jobs.
Basically, you need to choose the lesser of two evils, and I'm not sure if one less fast GC launch is worth what you are doing. You should consider this solution in the context of your application.
source share