Technology for a robust, persistent stack

Question

Mental reset attempt here: I tried to create a reliable, persistent stack with MSMQ, did not work

So, in more general terms:

I have a producer (web service, so multi-threaded, although "only one") / consumer (several processes, how many are needed). The key problems are - Data must be consumed / processed in LIFO order (~> stack) - Data must be saved / processed in a reliable way (that is, with support for disk, message queue, etc.). Bonus points for transaction support. - interaction between processes

Given the above points, I try my best to find a neat solution. What I looked at:

• Do it yourself It was not really planned to do it, but the initial proof of concepts for this only confirmed that it was difficult (for me) and helped me better understand many obstacles.

• MSMQ It would be nice and easy, as it easily lends itself to "reliable", is easy to configure and is already part of the target infrastructure. Unfortunately, "LIFO" / "Stack" is the killer here. It is impossible to do → Bzzzt.

• Database (SQL Server) I tried to take a look at the DB approach, but there are a lot of ugly things:

  • I need to save my data as blob (since it does not lend itself to column-based storage)
  • Polling a database for work just seems wrong (doesn't it?)
  • Blocking with multiple consumers seems complicated.

, ? , -, "" , / .

---

• Windows
• (.. /, , )
• , : /, . "", , . /, , : /?

Answer 1

SQL Server .

• , blob, ( , ). CREATE TABLE Stack (Id int identity, Data varbinary(MAX))

• . SQL Server , , , . SELECT * FROM Stack

• - , . ( ), , ( ) . , . , . # 2.

:

BEGIN TRANSACTION
SELECT Data FROM Stack WHERE Id = (SELECT MAX(Id) FROM Stack)
DELETE FROM Stack WHERE Id = (SELECT MAX(Id) FROM Stack)
COMMIT

, :

DELETE Stack
OUTPUT DELETED.Data
WHERE Id = (SELECT MAX(Id) FROM Stack)

10 , SQL :

DELETE Stack
OUTPUT DELETED.*
WHERE Id IN (SELECT TOP 10 Id FROM Stack ORDER BY Id DESC)

Answer 2

AMQP. google atm , , , , , FIFO LIFO , .

, , , .

Answer 3

DB, . , .

Answer 4

3 this SO Jon Skeet, , ....

- , Windows, WCF ? , ADO.NET ( MSDN), blog System.Transaction .

Answer 5

MapReduce , Google -. , , Hadoop