Since the audit object does not provide an earth-level attribute for the logical name / code type of the object to which a particular record belongs, it is best to associate the object (or entities) with you; you want to find audit records for - that is, without receiving all records.
A common technique for such scenarios is that you can refer to a related object with a half-saturation condition, for example, to verify that the primary key is not null. A link is enough for your case.
Example for pulling audit records tied to a contact:
from a in OrgContext.CreateQuery<Audit>() join c in ContactSet on a.ObjectId.Id equals c.ContactId where a.ObjectId != null && a.CreatedOn > since select a
source share