(?R) not a magic spell to get a pattern that can handle balanced things (like brackets). (?R) is the same as (?0) , it is an alias for the "capture group zero", in other words, the whole template.
In the same way you can use (?1) , (?2) , etc. as aliases for submatrices in groups 1, 2, etc.
Note that, with the exception of (?0) and (?R) , which, obviously, are always in their submatrix, since this is the whole pattern, (?1) , (?2) cause recursion only if they are in their Own own groups and can be used only in order not to rewrite part of the template.
something\((?:[^()]|(?R))*\) does not work, because for each nested (or not) opening bracket, something in your line precedes.
Conclusion, you cannot use (?R) here, and you need to create a capture group to handle only nested parentheses:
(\((?:[^()]|(?1))*\))
which can be recorded in a more efficient way:
(\([^()]*(?:(?1)[^()]*)*+\))
To finish, you need to add something that is no longer included in recursion:
something(\([^()]*(?:(?1)[^()]*)*+\))
Note that if something is a submatrix with an undefined number of capture groups, it is more convenient to refer to the last open capture group with a relative link, for example:
som(eth)ing(\([^()]*(?:(?-1)[^()]*)*+\))