In Caching To Scale Web Applications, we looked at ways to cache page response to url request by setting http cache headers to enable upstream systems to cache, mostly treating web application as black box. In this post we will look at
- Cache scopes
- Caches Policies
- Caching storage strategies and implications on performance
Web Application Caching
Caching within web application can be done at different scope and granularity.
Some well know (common) ways of caching in web application is based on scope of cached values
- Session Cache :– Cached value is stored per key per user and cached values are used for a single user. For example :– User home country.
- Application Cache :– Cached value is stored per key and cached values are used for multiple users. For example :– List of countries.
Session cache’s Time To Live (TTL) is usually in minutes while Application cache’s TTL varies widely and usually cached forever until changed with application events or business triggers.
Application cache is memory efficient compared to Session cache, as same cached values is used by multiple users in application cache. But if the cached value is personalized data for a specific user, it should stay in Session Cache.
While Session Cache has been used a lot (sometime abused), Application cache is not widely used.
Cache Policies
Write-through : Cache is updated along with backing datastore synchronously. Since both datastore and cache is always kept in sync, Write-through provides high data integrity and consistency at the cost of performance. Write-through caching makes sense with read heavy applications with very few writes.
Write-back : Cache is updated synchronously and backing datastore is updated asynchronously. Since only cache is updated synchronously Write-back provides better performance but at the cost of inconsistency or data loss in an event of crash. Write-back caching policy makes sense when there is large number of writes and lossing latest data does not effect application.
As you can see detecting and handling cached value modification is very important. In fact, it is good practice to make all cached values immutable.
Cache strategies
Below are different cache storage options available
- In-memory cache : cached values are stored in RAM memory.
- (a) in-process : caching with-in application process
- (b) out-of-process : caching in another process
- Persistent cache : caching in persistent systems like files or database.
To choose caching option, we have to understand performance characteristics of different storage option. Lets start with time taken to perform typical operations on computer. Below numbers are from Jeff Dean presentation at Google.
| Numbers Everyone Should Know: | |
|---|---|
| Execute typical instruction : | 1/1,000,000,000 sec = 1 nanosec |
| Fetch from L1 cache memory : | 0.5 nanosec |
| Branch mis-prediction : | 5 nanosec |
| Fetch from L2 cache memory : | 7 nanosec |
| Mutex lock/unlock : | 100 nanosec |
| Fetch from main memory : | 100 nanosec |
| Compress 1K bytes with Zippy : | 10,000 nanosec |
| Send 2K bytes over 1Gbps network : | 20,000 nanosec |
| Read 1MB sequentially from memory : | 250,000 nanosec |
| Round trip within same datacenter : | 500,000 nanosec |
| Fetch from new disk location (seek) : | 10,000,000 nanosec |
| Read 1 MB sequentially from network : | 10,000,000 nanosec |
| Read 1MB sequentially from disk : | 30,000,000 nanosec |
| Send packet CA->Netherlands->CA : | 150 millisec = 150,000,000 nanosec |
First thing to notice from above numbers is, L1 and L2 cache. Caching is not only used in applications, network software, database systems, operating systems but also in Computer design.
Back of the Envelope Calculations to determine performance of different caching strategies
In-memory and in-process caching :– Cache reads are fetched directly from current process memory.
In-memory and in-process cache fetch time = Fetch from main memory = 100 nanosec
In-memory and out-of-process caching :– Cache reads are fetched from another process usually over network. Cached values are stored in another processes memory.
In-memory and out-of-process cache fetch time = Round trip within same datacenter + Fetch from main memory = 500,000 nanosec + 100 nanosec
Persistent caching :– Cache reads are read from disk usually over network.
In-memory and out-of-process cache fetch time = Fetch from new disk location (seek) + Round trip within same datacenter = 10,000,000 nanosec + 500,000 nanosec
Persistent caching does not always mean reading from disk, databases usually cache working set data ( often used data ) in its main memory so when there is cache hit it performs similar to In-memory and out-of-process.
Summary
- In-memory and in-process caching is 500 times faster than In-memory and out-of-process cache fetch time
- In-memory and out-of-process cache fetch time is 20 times faster than Persistent caching
Clearly in-memory in-process caching is big winner and is widely used in web applications. Does that mean we should always use in-memory in-process caching ? To answer this question we have to understand characteristics of large scale web application.
Caching in Web Applications Cluster
As discussed in Architecture Issues Scaling Web Applications, large scale web application should be able to
Horizontal scale out We should be able to add identical nodes in each layer to scale web applications.
No single point of failure Is large cluster of nodes, failure of single node can happen and it should not bring down application.
Due to above two characteristics we end-up with cluster of nodes in any large scale applications.
Getting back to caching, if we go with in-memory in-process caching, each node will have to cache required data. Below are few issues with in-process caching
- Redundant caching consumes lot of memory. In cluster with N web application processes same cached value has to be stored N times.
- Each node will have to face cache miss and perform resource intensive operation before caching. i.e. In N Node cluster resource intensive operation is performed N times. This problem become noticeable if TTL of cached value is low. For Example :– A web application cluster with 100 nodes, and TTL of 1 min will perform 100 resource intensive operations every minute.
- Cache Refresh is another major problem with in-memory in process in cluster of web application process. Refreshing cache across the cluster is not easy and if cache is refreshed based on time. Due to machine time synchronization issue stale cached value will be used in some nodes, depending on application it could cause application in-consistent results based on which node handles request.
Out-of-process caching gets around the above issues by storing cached values in distributed caching systems like memcache. Due to central cache handling in out-of-process caching
- If one of the web application node caches a value, it is available to all other nodes reducing cache misses.
- Cache can be refreshed or invalided easily by updating central cache system.
Persistent caches is used when it is important to recover from crash with cached values intact. It is achieved by re-loading cached data from disk.