Random database errors with Django 1.7, uwsgi and PostgreSQL

Question

After upgrading my Django 1.6 application to Django 1.7, I started getting random errors when retrieving data from PostgreSQL:

DatabaseError: server sent data ("D" message) without prior row description ("T" message) lost synchronization with server: got message type " ", length -1244613424 DatabaseError: lost synchronization with server: got message type "0", length 842674226 ProgrammingError: no results to fetch ValueError: invalid literal for int() with base 10: 'feuj3f47jvsdv7tgnj43g63j' 

When I quickly open 10 tabs in the browser, half of the downloads usually load, half of them get a DB error. When I update tabs with errors, they load normally.

I am running Django through uwsgi and nginx, psycopg2 version is 2.5.4.

In general, it seems that some connection with Postgres is completely broken, and the results of different queries become mixed.

---

Edit:

After several hours of troubleshooting, the following came out:

Django 1.6 + uwsgi - works
Django 1.7 + gunicorn - works
Django 1.7 + uwsgi - does not work, causes database errors. So the problem is with the features of uwsgi and Django 1.7. What is strange, I have another Django 1.7 project running on the same server with the same uwsgi, and it has no problems.

Any ideas?

(In fact, I do not mind switching to guns, you may have to go this route, but still I wonder why this happens)

---

Update 2: a more detailed study reveals absolutely crazy things happening inside Django, for example, the primary key of the model, replaced by the current user session_id (which is the source of the "invalid literal for int () with a database error of 10") and Django issuing database queries " forgets "to indicate a WHERE clause. I would probably call it memory corruption.

---

Update 3: We switched from uwsgi to gunicorn and the problems are now gone. Everything works great. Maybe I'm still looking for a suitable solution.

Answer 1

I think lazy-apps=true should do the trick. From the uwsgi documentation:

uWSGI tries (ab) to use the Copy On Write semantics of fork () when possible. By default, it will work after your applications load as much of their memory as possible. If for some reason this behavior is undesirable, use the lazy application setting. This will instruct uWSGI to load applications after each working fork (). Beware, as there are older options called lazy that are more invasive and very discouraged (they are still here for backward compatibility only)

If you do not enable lazy-apps , workers will exchange memory and most likely ruin it:

the primary key of the model is replaced by the current user session_id.

When it comes to connections and stuff, installing lazy-apps dangerous.

The disadvantage is that every worker will have a full connection pool (if pooling occurs), and you can use many connections.

I'm not a python expert, but I think using something like gevent to centralize the connection pool. You may not even need lazy-apps .