This is almost the situation I am in, except that I use Knockout.js and not Backbone. I have views for Create and Edit and in each view there is a very complex Knockout.js user interface that loads Ajaxing JSON back to the server.
In MVC3, I used many JsonResult methods inside the same controller that displayed the views. I experimented with RC MVC4 and wondered if I should follow the "ho" path of using the API controller for Ajax requests. I like strongly typed HTTP classes and the fact that JSON.NET is more integrated, but at this point I have to say that the end result of having a separate API controller for my internal use was simply not right. Like you, I found that I had a lot of duplication around security, and the separation of what was connected with the logic simply by the type of content made things more confusing rather than clean.
So at the moment (although I was already known), I plan to use standard MVC controllers for my current context, but I can take the opportunity to use the brilliant new web API if I ultimately expose the public API.
source share