RAM and Cache

Register, Cache and RAM

Storage Hierachy

For a comprehensive understanding on why RAM is much slower than registers, read Mike Ash's explaination.

Different CPU architectures have different layout of cache.

Cache Layouts

Principles of Locality

Locality means data should move as little as possible! Programs tend to reuse data & instructions that are close to each other or they have used recently.

Temporal locality 1. Recently referenced items are likely to be referenced in near future 2. A block tends to be accessed again & again

Spatial locality 1. Items with nearby addresses tend to be referenced close together in time 2. Nearby blocks tend to be accessed Modern machines tend to read blocks of lower memory into the next level of the memory hierarchy. Following data locality makes the hierarchical memory layout profitable. No matter in hard disk or RAM, sequential access is faster than random access.

Cache Hit Rate

Suppose that the L1 cache has a 1 nanosecond access latency, a CPU must load the data from the L1 cache 100 times and the hit rate is 100%, it will take the CPU 100 nanoseconds to perform this operation.

Now if the cache hit rate is reduced by 1 percent to be 99%, one out of 100 reads will have to access the L2 cache, which has a 10-cycle (10 nanoseconds) access latency. As a result, it takes the CPU 109 nanoseconds to complete the task: 99 nanoseconds for the first 99 reads and 10 nanoseconds for the last read. One percent reduction in cache hit rate leads to a ten percent of increase in latency.

Keep in mind, the above example is assuming that the missed data sits in the L2 cache. If the data has been evicted from the cache but it sits in main memory, with an access latency of 80-120ns, the performance difference between a 95 and 97 percent hit rate could nearly double the total time needed to execute the code.

In the real world, an L1 cache typically has a hit rate between 95 and 97 percent, but the performance impact of those two values in our simple example isn't 2 percent — it's 14 percent. [TODO:what does this mean]