Filter
Top Products
Chapter 1
Overview
CPUs and Memories
45
- Cellmap (across cells)
- Link (across fabrics)
Memory Bank Attribute Table
The MBAT interleaving is done on a per-cell basis before the partition is rendezvoused. The cell map and
fabric interleaving are done after the partition has rendezvoused. SDRAM on the cell board is installed in
physical units called echelons. For the new sx2000, there will be 16 independent echelons. Each echlon
consists of two DDR DIMMs. Each rank can have multiple internal logical units called banks, and each bank
contains multiple rows and columns of memory. An interleaving algorithm is used to determine how a rank,
bank, row, or column address is formed for a particular physical address.
The 16 echelons in the memory subsystem can be subdivided as follows: Four independent memory quadrants
are accessed by four independent MID buses. Each quadrant contains two independent SDRAM buses. Four
echelons can be installed on each SDRAM bus. The CC contains four MBATs, one for each memory quadrant.
Each MBAT contains 8 sets of routing CSRs, or one per rank. Each routing CSR specifies the bits of the
address that are masked or compared to select the corresponding rank, referred to as interleave bits. The
routing CSR also specifies how the remaining address bits are routed to bank, row, and column address bits.
To optimize bandwidth, consecutive memory accesses are used to target echelons that are as far from each
other as possible. For this reason, the interleaving algorithm programs the MBATs so that consecutive
addresses target echelons in an order that skips first across quadrants, then across SDRAM buses, then
across echelons per SDRAM bus, then across banks per rank.
Cell Map
Cell mapping creates a scheme that is easy to implement in hardware and to enable calculation of the
interleaving parameters for software. In order to do this, part of the physical address performs a lookup into a
table which gives the actual physical cell and the ways of interleaving into memory at this address. In order to
accomplish this there are some constraints:
- A portion of memory that is being interleaved across must start at an offset that is a multiple of the memory
chunk for that entry. For example, to interleave across 16 GB of memory with one entry, the starting address
for this chunk must be 0 GB, 16 GB, 32 GB, 48 GB, and 64 GB. If using three 2 GB entries to interleave across
three cells, then the multiple must be 2 GB, not 6 GB.
- Interleaving is performed across the actual cells within the system. Interleaving may be done across a
minimum of 0.5 GB on a cell, and a maximum interleave across 256 GB per cell.
- Each cell in an interleave group must have the same amount of memory interleaved. That is, you cannot
interleave 2 GB in one cell and 4 GB in another cell.
Link Interleaving
The link interleaving functionality did not exist in the sx1000. This logic is new for the sx2000 CC. The
sx2000 allows cells to be connected through multiple paths. In particular, each CC chip has three crossbar
links. When one CC sends a packet to another CC, it must specify which link to use.
The cell controller chip (CC) of the sx2000 chipset interfaces to processors, main memory, the crossbar fabric,
an I/O subsystem and processor dependent hardware (PDH). Two data path cpu bus interfaces are
implemented, with support for up to four processors on each bus. The address bus is 50 bits wide, but only 44
bits are used by the CC. Error correction is provided on the data bus and parity protection is provided on the
address bus.