Question: memory layout multi proc. for Opteron systems??

Ed <nomail.TakeThisOut@hotmail.com> writes:

 > On 10 Nov 2003 23:26:37 -0500, Mannr.TakeThisOut@uwaterloo.ca wrote:
 >
  > >I've asked this before but I don't think I got an answer.
  > >
  > >What is the memory layout on dual Opteron systems?
  > >For example:
<font color=green>  > > <a style='text-decoration: underline;' href="http://www.tyan.com/products/html/thunderk8w.html</font" target="_blank">http://www.tyan.com/products/html/thunderk8w.html</font</a>>
  > >
  > >Do I put memory in both banks? Is there any OS function that controls
  > >what processor gets what (the "closest") memory? I am just running vanilla
  > >(32bit) Linux.
  > >
  > >Also, does anyone have "stream" benchmarks for this machine? I'm curious
  > >what improvement results from the onchip memory controller has, and if
  > >applicable, what degradation results from the hypertransport.
  > >
  > >Thanks for any info,
  > > Richard
 >
 > Download the mother board manual, (Acrobat PDF file)
<font color=purple> > <a style='text-decoration: underline;' href="http://www.tyan.com/support/html/manuals.html</font" target="_blank">http://www.tyan.com/support/html/manuals.html</font</a>>
 >
 > Thunder K8W, section 2.07 - Installing The Memory.(pages 19-22)
 >
 > Ed

Thanks for the information. I just checked. OK I guess the chips will access
memory more or less transparently, no matter where it is.

But my questions still remain:

- What is the performance of accessing memory directly and going through
hypertransport? I would like to see both numbers, since I expect
direct memory accesses to improve due the onchip controller too.
I guess it is possible that hypertransport adds very little to the
access time, compared to the memory itself. But I'd like to see
some numbers.

- Will Linux (or any other OS) make sensible decisions, eg., allocating
new memory on the CPU it is running on, and, if possible, keeping the
thread/process on that processor in the future? If so, this architecture
could scale very well.

Richard<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems?? 

Mannr.RemoveThis@uwaterloo.ca wrote:
 > Ed <nomail.RemoveThis@hotmail.com> writes:
 >
 >
  >>On 10 Nov 2003 23:26:37 -0500, Mannr.RemoveThis@uwaterloo.ca wrote:
  >>
  >>
   >>>I've asked this before but I don't think I got an answer.
   >>>
   >>>What is the memory layout on dual Opteron systems?
   >>>For example:
<font color=brown>   >>> <a style='text-decoration: underline;' href="http://www.tyan.com/products/html/thunderk8w.html</font" target="_blank">http://www.tyan.com/products/html/thunderk8w.html</font</a>>
   >>>
   >>>Do I put memory in both banks? Is there any OS function that controls
   >>>what processor gets what (the "closest") memory? I am just running vanilla
   >>>(32bit) Linux.
   >>>
   >>>Also, does anyone have "stream" benchmarks for this machine? I'm curious
   >>>what improvement results from the onchip memory controller has, and if
   >>>applicable, what degradation results from the hypertransport.
   >>>
   >>>Thanks for any info,
   >>> Richard
  >>
  >>Download the mother board manual, (Acrobat PDF file)
  >>http://www.tyan.com/support/html/manuals.html
  >>
  >>Thunder K8W, section 2.07 - Installing The Memory.(pages 19-22)
  >>
  >>Ed
 >
 >
 > Thanks for the information. I just checked. OK I guess the chips will access
 > memory more or less transparently, no matter where it is.
 >
 > But my questions still remain:
 >
 > - What is the performance of accessing memory directly and going through
 > hypertransport? I would like to see both numbers, since I expect
 > direct memory accesses to improve due the onchip controller too.
 > I guess it is possible that hypertransport adds very little to the
 > access time, compared to the memory itself. But I'd like to see
 > some numbers.
 >
 > - Will Linux (or any other OS) make sensible decisions, eg., allocating
 > new memory on the CPU it is running on, and, if possible, keeping the
 > thread/process on that processor in the future? If so, this architecture
 > could scale very well.
 >

There is a PDF at AMD that, among other things, discusses your last
question. Basically, by default, when a *proccesor* needs memory,
it will try to allocate from the banks "attached" to it before it uses
memory "attached" to other processors.

However, it is *possible* for someone to create an OS that micro-manages
the memory allocations instead of letting the processors decide. AFAIK,
Linux and 64 bit Windows will *not* override the default. 32 bit versions
of Windows pretty much had their feature sets fixed before this became
something for MicroSoft to worry about, so they will or course simply let
the processors take care of this.

And yes, this architecture scales *very* well. Scaling from one
to 2, or 2 to 4 processors with Opterons give *much* better results
than the same kind of scaling with Itanic or Xeon. Opty vs Xeon
benchmarks that show the benefits of Opterons scaling are easy
to find on the web, while Opty vs Itanic takes a lot more searching.

Note also that AMD left the door open for any chipset/motherboard
manufacturer who wants to provide their own memory controller instead
of using the ones in the processors. AFAIK, no one has done this yet.



--
Reply to newsgroup only please. This e-mail account is real
but effectively abandoned because of excessive spamming.<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems?? 

Mannr.TakeThisOut@uwaterloo.ca wrote:
 > I've asked this before but I don't think I got an answer.

 > What is the memory layout on dual Opteron systems?
 > For example:
<font color=purple> > <a style='text-decoration: underline;' href="http://www.tyan.com/products/html/thunderk8w.html</font" target="_blank">http://www.tyan.com/products/html/thunderk8w.html</font</a>>

 > Do I put memory in both banks?

Yes. So you will need 4 DIMMs (2 banks, each 128 bit wide,
so 2 DIMMs per bank).

 > Is there any OS function that controls
 > what processor gets what (the "closest") memory?

Check the links found by:
<a style='text-decoration: underline;' href="http://www.google.com/search?q=linux+NUMA" target="_blank">http://www.google.com/search?q=linux+NUMA</a>

(NUMA = Non-Uniform Memory Access)

Linux Scalability Effort: NUMA Group Homepage
Linux Support for NUMA Hardware. Large ... This page provides links to information
about the various Linux on NUMA projects. Discussions ...
lse.sourceforge.net/numa/ - 7k - Cached - Similar pages

Linux: NUMA Awareness Added To Scheduler
.... Linux: NUMA Awareness Added To Scheduler. Posted by jeremy on Wednesday,
January 22, 2003 - 05:22. After several earlier attempts ...
kerneltrap.org/node/view/562 - 33k - Cached - Similar pages

The second one seems to show that the Linux Scheduler is aware of
NUMA (probably only the bleeding-edge versions).

 > I am just running vanilla (32bit) Linux.

Suse 9.0 64 bit is out now.
SuSE Linux 9.0 Professional 64 Bit Edition
<a style='text-decoration: underline;' href="http://www.amazon.co.uk/exec/obidos/ASIN/B0000UI2WS/" target="_blank">http://www.amazon.co.uk/exec/obidos/ASIN/B0000UI2WS/</a>

<a style='text-decoration: underline;' href="http://shop.mensys.nl/catalogue/mns_SuSELinux.html" target="_blank">http://shop.mensys.nl/catalogue/mns_SuSELinux.html</a>

I am waiting for an Opteron processor (and memory) to test it (all
the other hardware is already in the shop). Maybe today or tomorrow.

 > Also, does anyone have "stream" benchmarks for this machine?

<a style='text-decoration: underline;' href="http://www.cs.virginia.edu/stream/#PeeCeeResults" target="_blank">http://www.cs.virginia.edu/stream/#PeeCeeResults</a>
I think there are no Opteron results there (or I didn't find them).
But the code is easy to compile and run, so you can do it yourself.

Some days ago I found this:

<a style='text-decoration: underline;' href="http://wwwseminars.web.cern.ch/wwwseminars/2003/2003-OtherFormats/t-20030903.ppt" target="_blank">http://wwwseminars.web.cern.ch/wwwseminars/2003/2003-OtherFormats/t-20030903.ppt</a>

In slide 13 there is:
1x Stream: 2x Stream: 4x Stream:
2x Opteron, 1.8 GHz,
HyperTransport: 1006 1671 MB/s 975 1178 MB/s 924 1133 MB/s
2x Xeon, 2.4 GHz,
400 MHz FSB: 1202 1404 MB/s 561 785 MB/s 365 753 MB/s

I found these numbers a bit suspect, because I don't know if the benchmark
works well when 2 copies or more are run at the same time.

 > I'm curious what improvement results from the onchip memory controller has,

For comparision, results from a P4 2.8 GHz, with an Intel 875 chipset.

Function Rate (MB/s) RMS time Min time Max time
Copy: 2666.6667 0.0802 0.0600 0.0900
Scale: 2666.6667 0.0792 0.0600 0.1000
Add: 3000.0000 0.1011 0.0800 0.1200
Triad: 3000.0000 0.1022 0.0800 0.1300

 > and if applicable, what degradation results from the hypertransport.

If you have the time, you could run it with both banks filled and only
one. But as STREAM reports the best time, it might not show anything
interesting.

--
<a style='text-decoration: underline;' href="http://www.mat.uc.pt/~rps/" target="_blank">http://www.mat.uc.pt/~rps/</a>

..pt is Portugal| `Whom the gods love die young'-Menander (342-292 BC)
Europe | Villeneuve 50-82, Toivonen 56-86, Senna 60-94<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems?? 

"Rob Stow" <rob.stow.DeleteThis@sk.sympatico.ca> wrote in message
news:vr185t9f7n9g82@corp.supernews.com...
 > Mannr.DeleteThis@uwaterloo.ca wrote:
 >
 > And yes, this architecture scales *very* well. Scaling from one
 > to 2, or 2 to 4 processors with Opterons give *much* better results
 > than the same kind of scaling with Itanic or Xeon. Opty vs Xeon
 > benchmarks that show the benefits of Opterons scaling are easy
 > to find on the web, while Opty vs Itanic takes a lot more searching.

_Generally_ correct. However...

The reason that the Opteron (generally) scales better than Xeon etc. is
that the Opteron has a NUMA memory structure, with each processor having
its own memory, while MP Xeons have one central shared memory. Let me
emphasize that I definitely prefer the Opteron's memory structure.
However, there are some algorithms/applications, including some
commercially important applications, where a shared memory SMP system is
best.

So the answer to the question of whether the Opteron or Xeon MPs are
better is, "usually the Opteron is better, but sometimes the Xeon is
better."<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems?? 

<Mannr.RemoveThis@uwaterloo.ca> wrote in message
news:yua3ccv78fb.fsf@tapir.uwaterloo.ca...
 > - What is the performance of accessing memory directly and going through
 > hypertransport? I would like to see both numbers, since I expect
 > direct memory accesses to improve due the onchip controller too.
 > I guess it is possible that hypertransport adds very little to the
 > access time, compared to the memory itself. But I'd like to see
 > some numbers.

Well, I think others have already provided the numbers. One thing to keep in
mind is that Hypertransport is so fast that AMD doesn't want developers to
spend an inordinate amount of time taking NUMA effects into account. They
have invented a new moniker they call SUMO (Sufficiently Uniform Memory
Organization, or something like that). I think they claim that accessing
memory through the Hypertransport link (HTL) is faster than the traditional
shared bus memory controllers anyways, because HTL doesn't have any bus
contention issues (they are all dedicated point-to-point links). So worrying
about optimizing an OS kernel is without merit, at least when it comes to
processors connected via HTL. I think the original idea of HTL was for it to
be the traditional full front side bus of the processor, including the link
to the external memory controller, but when they decided to integrate the
memory controller into the processor, then the HTL was freed up to become
simply the link to the I/O chips and the other Opterons.

They would rather that NUMA tuning efforts be geared towards external
interconnects. For example, with HTL one could place upto 8 Opterons in a
SUMO configuration inside a single systemboard, but once you want to use
more than 8 Opterons, you have to add additional system boards, and connect
each of the systemboards to each other via some interconnect other than HTL.
So they suggest treating SUMO simply like SMP, while slower systemboard to
systemboard interconnects can be treated like NUMA.

 > - Will Linux (or any other OS) make sensible decisions, eg., allocating
 > new memory on the CPU it is running on, and, if possible, keeping the
 > thread/process on that processor in the future? If so, this
architecture
 > could scale very well.

I think they've already discovered that the architecture can scale extremely
well, even without taking NUMA into account.

Yousuf Khan<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems?? 

Rui Pedro Mendes Salgueiro <rps RemoveThis @rena.mat.uc.pt> writes:

 > Mannr RemoveThis @uwaterloo.ca wrote:
  > > I've asked this before but I don't think I got an answer.
 >
  > > What is the memory layout on dual Opteron systems?
  > > For example:
<font color=green>  > > <a style='text-decoration: underline;' href="http://www.tyan.com/products/html/thunderk8w.html</font" target="_blank">http://www.tyan.com/products/html/thunderk8w.html</font</a>>
 >
  > > Do I put memory in both banks?
 >
 > Yes. So you will need 4 DIMMs (2 banks, each 128 bit wide,
 > so 2 DIMMs per bank).
 >
  > > Is there any OS function that controls
  > > what processor gets what (the "closest") memory?
 >
 > Check the links found by:
<font color=purple> > <a style='text-decoration: underline;' href="http://www.google.com/search?q=linux+NUMA</font" target="_blank">http://www.google.com/search?q=linux+NUMA</font</a>>
 >
 > (NUMA = Non-Uniform Memory Access)
 >
 > Linux Scalability Effort: NUMA Group Homepage
 > Linux Support for NUMA Hardware. Large ... This page provides links to information
 > about the various Linux on NUMA projects. Discussions ...
 > lse.sourceforge.net/numa/ - 7k - Cached - Similar pages
 >
 > Linux: NUMA Awareness Added To Scheduler
 > ... Linux: NUMA Awareness Added To Scheduler. Posted by jeremy on Wednesday,
 > January 22, 2003 - 05:22. After several earlier attempts ...
 > kerneltrap.org/node/view/562 - 33k - Cached - Similar pages
 >
 > The second one seems to show that the Linux Scheduler is aware of
 > NUMA (probably only the bleeding-edge versions).
 >
  > > I am just running vanilla (32bit) Linux.
 >
 > Suse 9.0 64 bit is out now.
 > SuSE Linux 9.0 Professional 64 Bit Edition
<font color=purple> > <a style='text-decoration: underline;' href="http://www.amazon.co.uk/exec/obidos/ASIN/B0000UI2WS/</font" target="_blank">http://www.amazon.co.uk/exec/obidos/ASIN/B0000UI2WS/</font</a>>
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://shop.mensys.nl/catalogue/mns_SuSELinux.html</font" target="_blank">http://shop.mensys.nl/catalogue/mns_SuSELinux.html</font</a>>
 >
 > I am waiting for an Opteron processor (and memory) to test it (all
 > the other hardware is already in the shop). Maybe today or tomorrow.
 >
  > > Also, does anyone have "stream" benchmarks for this machine?
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://www.cs.virginia.edu/stream/#PeeCeeResults</font" target="_blank">http://www.cs.virginia.edu/stream/#PeeCeeResults</font</a>>
 > I think there are no Opteron results there (or I didn't find them).
 > But the code is easy to compile and run, so you can do it yourself.
 >
 > Some days ago I found this:
 >
<font color=purple> > <a style='text-decoration: underline;' href="http://wwwseminars.web.cern.ch/wwwseminars/2003/2003-OtherFormats/t-20030903.ppt</font" target="_blank">http://wwwseminars.web.cern.ch/wwwseminars/2003/2003-OtherFormats/t-20...903.ppt</a>>
 >
 > In slide 13 there is:
 > 1x Stream: 2x Stream: 4x Stream:
 > 2x Opteron, 1.8 GHz,
 > HyperTransport: 1006 1671 MB/s 975 1178 MB/s 924 1133 MB/s
 > 2x Xeon, 2.4 GHz,
 > 400 MHz FSB: 1202 1404 MB/s 561 785 MB/s 365 753 MB/s
 >
 > I found these numbers a bit suspect, because I don't know if the benchmark
 > works well when 2 copies or more are run at the same time.
 >
  > > I'm curious what improvement results from the onchip memory controller has,
 >
 > For comparision, results from a P4 2.8 GHz, with an Intel 875 chipset.
 >
 > Function Rate (MB/s) RMS time Min time Max time
 > Copy: 2666.6667 0.0802 0.0600 0.0900
 > Scale: 2666.6667 0.0792 0.0600 0.1000
 > Add: 3000.0000 0.1011 0.0800 0.1200
 > Triad: 3000.0000 0.1022 0.0800 0.1300
 >
  > > and if applicable, what degradation results from the hypertransport.
 >
 > If you have the time, you could run it with both banks filled and only
 > one. But as STREAM reports the best time, it might not show anything
 > interesting.
 >
 > --
<font color=purple> > <a style='text-decoration: underline;' href="http://www.mat.uc.pt/~rps/</font" target="_blank">http://www.mat.uc.pt/~rps/</font</a>>
 >
 > .pt is Portugal| `Whom the gods love die young'-Menander (342-292 BC)
 > Europe | Villeneuve 50-82, Toivonen 56-86, Senna 60-94

Thanks for the detailed information!<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Question: memory layout multi proc. for Opteron systems??