How does a typical medium or large web application “grow” in terms of infrastructure?

Question

Skinny : when Facebook / Twitter / Youtube (independently) switched from a basic idea to software ... more (maybe 100,000 users?), How did they grow?

Is there a way to improve “best practices” for midsized web applications?

The real question . When you quote or bid on a mid-sized web application project, what is biggies? In this case, we will use the PHP framework, but it seems that they will mainly be generalized to any language.

Thus, programmers for the main application (for me) are the most obvious part. We get user control, user interface and special classes for processing the application. However, it seems to me that this is less than half of the real project.

Ultimately, with good growth, infrastructure and meta-interface issues will be your main focus, right?

1) Infrastructure: application cloud space, data storage, db synchronization for situations with multiple data centers.

2) Problems of language and culture: the creation of the application seems “pretty” or at least useful in the main “cultural markets”

3) problems with indexing data

4) API / compatibility issues (both embedded ala facebook applications and external data access for both end users and large players such as search engines, etc.)

... so I'm so sure that about half of them are missing, and I don’t know how they prioritize.

The accepted answer here is a pretty good starting point for the answer I'm looking for.

Answer 1

Expanding the number one position on your list is likely to be quite key and will help you decide which scaling problems you have, and even understand which classes of technologies can be useful. This will also apply to paragraphs 3 and 4, as they are interrelated.

Below is a big question of questions that can help you in your search mind expand your thoughts on scalability, and this is not a direct answer to your question, but hopefully a starting point:

What are your features in the app? Do they read heavy or heavy, or maybe both? When you browse data, should it always stay in real time? OR may be delayed? How far can it be delayed? Can caching help? Caching is the easy part, also think about how EXPIRE caches are where the hard stuff is. What does the data you work with look like? Is it very relational or more like individual documents?

What are your performance requirements? - Basically, the application creates reports in the background and sends them by email? Or does he need to display a perfect map in real time of all current tweets from twitter as they arise? Do I need to instantly distribute updates from users to other users? Every user or just a subset of them? How fast should this be done? Should a page load before 300 ms or less than 2 seconds?

Do external services have any restrictions, possibly access restrictions or bid limits? This means that you will need to stand in line and order inquiries. Some of your external data providers send you data faster than you can process it? You may need to queue, you may need to make this part of the system scalable on its own, with a variable number of “workers” that can move up and down. Consider applying this “individually scalable” principal to other parts of the system; it pays dividends in large installations.

Hope this helps a bit :)