What does Twitter and Scala middleware mean?

Question

Link is another SO question . I was given in this article about Twitter migrating from Rails to Scala, and the article submitted this comment:

By the end of this year, Payne said: Twitter hopes the entire middleware infrastructure and its API are ported to a new language. Ruby will remain, but only at the front end. "We're still happy with Rails for building custom functions ... in terms of performance, it's good for people to click on web pages. It's hard work, asynchronous processing is the type of material that we moved away from."

What does middleware mean here? What exactly does it mean that Ruby is on the front side? Does this mean that the rails on the front are used with a very small ORM? How does he "meet" with scala? What does this comment mean?

Just trying to understand architecture. Thanks.

Answer 1

This is a form of N-tier architecture . Initially, most sites will begin as a 2-tier architecture, with a web server and database. The web server will serve the pages that the user sees, while these dynamic pages access the database.

When the architecture is divided into several levels, usually you will have a web application in front, with some kind of application server that will invoke the web application. This application server contains business logic and makes the application functional. The “face” for the application simply makes things look beautiful and formats it for display to users.

An application server or “middleware” is simply a collection of functions that can be called by web servers. Middleware should be powerful for a site such as twitter, where most of the activity when sending messages, and from people accessing the website, is not so much.

The interconnection between the tiers is probably the standard web service technology, but it could be the usual thing, perhaps REST or some other kind of web service where the web application (Ruby on Rails) can access the application data.

In this type of architecture, Rails certainly categorized all ORM and data management as middleware.

Middleware is also an indefinite term and can consist of many levels, which is why they are called N-Tier. I would argue that Twitter has another part of its messaging middleware, while the other part is about managing accounts, receiving messages, etc.

The idea is to be able to scale up by adding hardware, where you can have a cluster of servers serving web pages, another messaging with the cluster, and then a cluster of database servers that supports all of this. This is not an exact science, each architecture is different, but, as a rule, you can think about it.

Answer 2

Try Google for " ruby twitter scala payne " and you will get more than you probably need on this.

My (perhaps incomplete) understanding is that most of the "back-end" of Twitter is not encoded particularly well (see the post by Obie Fernandez who selects some quotes from Alex Payne). This, at least in part, is due to the fact that the initial fast-growing nature of the application, which was overtaken by events, they experienced significant pain, as the use of the application began to grow exponentially. At that time, there were a few fingers pointing to Twitter (and the Rails association) for the “inability to scale” at the time.

Twitter is very dependent on the message queue, and this part that I understand was written in Scala, which is probably (without personal experience) a very good choice of language to work with.

Answer 3

I understand that what they transferred to Scala is the message queue: from Ruby-ORM-DB backup to a distributed messaging system using Scala. The Twitter volume service has special requirements, and they created their own messaging solution. Called "kestrel"

Answer 4

Messaging service in this case. They probably switched from some Ruby-based custom solution to JMS.