Assuming you are creating a simple (game) billiard simulator, you can use something like:
v_new = coeff*(v_old - 2*dot(v_old, boundary_normal)*boundary_normal);
Here v_old is your current velocity vector, and boundary_normal is the inwardly normal circle of your pool table at the point of impact. If you know the center c your pivot table and you have a hit point p , then the normal is just normalize(cp) . That is, the normalized vector that you get when subtracting p from c .
Now I took coeff as a leaching factor between 0 (no speed more after a hit) and 1 (same speed after a hit). You can make this more plausible by determining the correct restitution coefficient.
In the end, all the formulas above are simply reflection , as you could see, for example, in the base ray. As said, this is a rather crude abstraction from the exact modeling of physics, but is likely to cope with this task.
source share