Data Caching Methods / Tips / AppFabric

Question

We have millions and millions of records in the SQL table, and we run very sophisticated analytics for this data to create reports.

As the table grows and additional records are added, the calculation time increases, and the user has to wait a long time before the web page loads.

We thought of using a distributed cache, such as AppFabric, to load data into memory when the application loads, and then run our reports from that data in memory. This will slightly improve the response time, since now the data is stored in memory and disk.

Before we take the plundge and implement this, I wanted to check and find out what others are doing, and which of the best methods and practices to load data into memory, caching, etc. Of course, you are not just loading the entire table with 100 million million records in memory ...

I also studied OLAP / data storage, which could give us better performance rather than caching.

Answer 1

A complex reporting solution is a preliminary calculation, so you are on the right track if you are looking at OLAP.

Answer 2

Have you considered database breakdown? We do this for our largest databases.

, , - , , IO.

Answer 3

SQL,

. .

.

SQL.

,

. .

, AppFabric...

. , RDBMS.

OLAP/

. . . , SQL - ( ) .