TACC Thoughts

August 14, 2008

This summer I visited the Texas Advanced Computing Center for a week-long Supercomputing Institute. The material presented during the institute was slanted towards the introductory, but it was a good trip. The MPI and optimization sessions were quite interesting, and Austin is a wonderful city. TACC’s marquee system is a NSF Cyberinfrastructure “Track 2” system named Ranger. Much like our group’s fastest cluster, it’s a quad-socket, quad-core AMD Opteron machine running CentOS (specifically, Rocks 4.2.1. Unlike our system, they have 62,976 CPU cores instead of 256. You’d think this would be a cabling nightmare. Nope! Check it out:

Sun Constellation Infiniband Interconnect

Sun Constellation Infiniband Interconnect

Rear View of Ranger Rack

Rear View of Ranger Rack

Pretty impressive. In the first picture, you see the two Sun Constellation Infiniband switches, back-to-back, with the longhorns on top. If not for each of those switches having over 3,000 ports, the system would require many, many more individual switches to link all the nodes together. In the background you can see the front side of invididual racks, each of which contain 4 rows of 12 nodes each (with cooling stacks inbetween compute racks).

More interestingly, in the second picture you can see what the cabling looks like behind the racks. On the right in that photo is a typical 48-node compute rack. Only 4 Infiniband cables are required per 12 nodes, because each cable contains 3 connects! Nice. They dont have any extra gig-E cables getting in the way, nor KVM cables sitting around. The power cables route nicely into the sides. The rack on the left is a bit more crowded: it’s a disk storage rack. Still, the cables aren’t a mess. Check out all the hot-swappable power supplies in both racks!

Adding up the money spent on building construction, initial hardware, operations and support, Ranger’s expense is roughly $50,000 per day over an expected 4 year lifespan. The air conditioning alone eats over a megawatt of power. You know, an Alaskan supercomputing center with an innovative building design for natural A/C could be a Really Good Idea.

What would you do with tens of thousands of cores? The most popular applications are computational fluid dynamics (physics, engineering) and molecular dynamics (chemistry, computational biology). In a few years, writers of desktop applications are going to be facing this problem on a much smaller scale, as new machines are sold with 8-16 cores standard.

I think the most promising future applications are in optimization and artificial intelligence. They may have unusually productive weak scaling properties. In the meantime, the best way to get a grant for these machines is to have a problem (or rather, a solution!) which involves a strong scaling calculation. Fortunately, our simulation codes fit the bill.

I have to wonder what they will do with Ranger in ten years…or less. Decommissioning a machine of that size is a large undertaking. Would it be sold to recyclers?

The TACC staff are quite nice and energetic. It’s clear they’re seeking out people with good ideas for scientific problems that need ten-thousand-core solutions. If they ever host a conference or workshop you’re interested in, it’s worth the trip. Rent a car to see Austin, and avoid my folly of visiting the week it’s over 100 degrees outside every day! Hah.


Completing the Cycle: Fresh Boot Camp with Windows XP SP3

May 24, 2008

I haven’t had a post about Windows yet, so here you go. My MacBook Pro has been dual-booting into Windows XP SP2 since I had OS X 10.4, “Tiger” installed. This is an older version of Boot Camp, with drivers that are now outdated. I also tinkered with them a bit by installing some newer NVidia drivers from laptopvideo2go.com.

I wanted a fresh Windows install with SP3 and the latest Boot Camp 2.1 drivers. Apple recommends that you install Boot Camp 2.1 before installing SP3. But what if you have a Windows disc with SP3 already slipstreamed? It should work, but I couldn’t find a good answer on the web, and decided to try it myself.

I created a slipstreamed SP3 disc from our University’s volume licensed SP2 disc using nLite. No problem. I then repartitioned my MacBook Pro’s drive using the Boot Camp Assistant and followed Apple’s instructions to the letter. The Windows install, using my SP3 disc, went fine. I did get a black screen after the initial reboot, but manually power cycling the machine gave a clean boot.

I proceeded to install the Boot Camp 2.0 drivers from the Leopard disc. This went mostly smoothly, but the “SM Bus Contoller” was not found in Windows Device Manager. To fix that problem, I re-inserted the Leopard DVD and told it to “repair” the Boot Camp installation. Maddeningly, this robotically proceeded to reinstall every Boot Camp driver. But hey, it fixed the problem! I then used Apple Software Update (which itself had to be updated) to install the Boot Camp 2.1 drivers. You know, the package Apple says should be installed before SP3 — that is true if you’re upgrading from SP2 to SP3. The Boot Camp upgrade took a while, but succeeded. For those who are curious, the NVidia drivers in Boot Camp 2.1 are version 167.73. I still had to run Windows Update to get IE7, WMP11, and a couple of hotfixes on top of SP3, but that’s nothing new under the Sun.

In summary, a Windows XP SP3 slipstreamed disc may successfully be used for a fresh install with Boot Camp.

It’s the little things.

April 1, 2008

The Usenet, once the dominant form of internet communication, has been rendered largely irrelevant by the World Wide Web. Yet, it can still be a great source of information: some technical groups (such as comp.lang.fortran) have an impressively high signal to noise ratio.

It was a search for esoteric Emacs knowledge that sent me to gnu.emacs.help. Emacs has a nice syntax highlighting ability which makes source code more readable, by associating color with functionality. I wanted intrinsic procedures such as “mod” or “exp” to highlight in a different color than executable constructs such as “if … then” statements. Fortran mode in Emacs does this by default, but F90 mode does not. Here’s an example:

Fortran mode

if (mod(k,10).eq.0) then
things to do

F90 mode

if (mod(k,10).eq.0) then
things to do

It’s a small difference, but in complicated code with many different procedures or constructs, it makes life much easier on my eyes. It wasn’t clear from the Emacs source how to make F90 mode match Fortran mode.

Luckily, Glenn Morris, the maintainer of the fortran.el and f90.el filters, reads gnu.emacs.help and was able to give me a fix:

I don’t really know why it is like it is, but it is. Anyway, add to~/.emacs:

(eval-after-load "f90"
     (setq f90-font-lock-keywords-3
      (append f90-font-lock-keywords-2
           (list f90-procedures-re '(1 font-lock-builtin-face keep))
      f90-font-lock-keywords-4 (append f90-font-lock-keywords-3
          (list f90-hpf-keywords-re)))))

It works — my Fortran 90 looks good again. So many parentheses! The recently deceased Arthur C. Clarke may have imagined that most people would categorize Emacs Lisp scripting as magic.