Let's be realistic: Wouldn't it be nice if we could make the Macs we already have (and don't use) do something useful for us?
While you may spend most of the time doing things for which a desktop Mac will suffice and even get bored with, the few exceptions cause those delays you probably get bored with too. So we find ourselves many times each day sitting in front of a progress bar, pretending not to notice a colleague's unused Mac. Why tolerate this waste?
Fortunately, we may soon no longer have to. The dream of being able to join forces with other Macs and draw from their compound capabilities is coming true on Mac OS X today. It's grid computing's debut on personal desktops.
Licence to grid
What used to be the privilege only of well-heeled research labs with deep pockets, full-time administrators and supercomputers, is finding its way to small businesses and home offices with OS X applications becoming increasingly capable of parallel computing.
Apple's Shake 3.5 (high-end video compositing and special effects software) for OS X ships as standard with a grid engine and a licence for unlimited rendering. Compressor, Apple's powerful video compression and encoding software that comes with Final Cut Pro and DVD Studio Pro, is also capable of distributing jobs across networks.
In the System Preferences pane, Shake Qmaster users can set up Unix Processing and cluster control services. You set up clusters in Shake Qadministrator, add Shake, Maya and Shell command jobs in Shake Qmaster and monitor them in Batch Monitor. Users of Final Cut Pro or DVD Studio Pro can use Compressor's clustering abilities for audio and video batch compression.
A network for parallel computing
There are several ways to setup a cluster. Most of them have some things in common, but you do need:
- Several computers (which you can network using Gigabit ethernet; Fibre Channel or InfiniBand).
- Good quality cables.
It is of course possible to cluster using an existing network. The biggest obstacle to setting up and running a cluster wasn't connectivity, but the complex software it required. A key technology widely deployed in parallel computing is MPI (Message-Passing Interface), an industry-standard. One implementation of MPI is mpich, a piece of software for which set-up is a real big-brainer, partly due to its use of NFS (network file system). Mpich requires countless complex set-up files be correctly configured to work, and this is an awkward, time-consuming process. With Mac OS X Server 10.3, the initial configuration is somewhat easier than on Linux or Solaris, but maintenance has its problems. You need to know a lot about NFS. Furthermore, you need rsh (remote shell) or its secure counterpart ssh (secure shell) to establish communications between nodes. Sure, you can deploy parallel computing this way, but if you have other things to do, you probably won't want too.
A fully-featured hassle-free environment for parallel computing on OS X is already exists: Pooch (Dauger Research). However, Pooch offers very powerful capabilities, but isn't yet up to the task of maintenance and manageability. An uninstaller utility ? or even better, a graphical activation and de-activation system, would save administrators vast amounts of time.
While Unix geeks will easily get handle various start scripts in .Pooch folders, doing it in the command line is no fun and the average user will struggle to manage this successfully. So if you wanted to switch between a "Pooch-enabled" and a "no-Pooch" environment on a daily basis, for example to turn a network of Power Macs into a grid for nightly computation, you would probably opt for dual-boot set-ups with netbooting. In a netboot-enabled environment you can easily choose a Pooch-enabled or Pooch-disabled environment, without having to provide two boot partitions on every computer involved in the process. An important advantage of the netboot idea is that you avoid reconfiguring any machine when setting up your ad-hoc cluster.
But these solutions still fall short of grid computing's promise.
Grid-enabling the OS
The purpose of grid computing is to join forces using otherwise independent machines which are not necessarily dedicated to the task, in order to compute results in otherwise impossible-to-achieve time frames. Grid software should be able to deploy the computing potential of both dedicated clusters and temporarily idle computers and load-balance the interconnected units without an administrator having to interfere.
Good news is coming. Apple is developing a truly user-savvy, nearly self-administering grid solution called Xgrid.
Xgrid uses Rendezvous to discover available resources. It can distribute, accept or reject tasks according to a set performance policy, which allows networked Macs to suspend remotely issued commands when local processes request runtime performance.
"One of the biggest benefits of Xgrid is that many applications can take advantage of it without any coding whatsoever," said Dr Richard Crandall, of Apple's Advanced Computation Group.
"Xgrid can be used 'out of the box' with virtually any tool that runs on the UNIX command-line. Simply type xgrid -job submit 'command' and Xgrid handles the hard work of uploading files, distributing tasks, and retrieving the results," he explained.
Xgrid is currently available for Panther as Technology Preview 2, and will be bundled as part of Mac OS X 10.4 "Tiger", both server and client.
"Xgrid has been a notable success, generating enormous amounts of interest in academic, government and commercial research organizations around the world, regardless of its Technology Preview status," said Crandall.
There is even more to come. Speaking to Macworld UK, Dr Crandall did not rule out the possibility of Apple adding grid capabilities to future versions of its desktop software.
Apple laid the groundwork for third-party developers to embed Xgrid client functionality into their own standalone applications using message-passing interface (MPI) for inter-node communication.
In future this means we conceivably could expect an application like iMovie to draw rendering results from the "grid" of other Macs running less time-consuming jobs on its network at any given moment.
"We are designing Xgrid to support a wide range of applications, from scientific computation to multimedia workflows," said Crandall. "As for scientific matters, Xgrid is intended for Mac users who need distributed computations as part of their normal research. An example: massive, 3-dimensional FFT (Fast Fourier Transformation), where one needs in the first place to fit say many gigabytes of original data by distributing said data across many machines and then use the same machine cluster to speed up the actual arithmetic."
Handily, Apple will introduce its Xsan storage area network software to support distributed systems this autumn.
Wherever problems can be solved by exploiting the number-crunching power of large-scale computers, grid technology is on everyone's lips. But Apple is the first to put this idea to broad use. Fighting its not-so-expanding share in the desktop market, Apple boldly put a foot in the server, cluster and SAN business with Xserves.
Apple has all that it takes to succeed here: a ground-breaking gigaflop-per-dollar ratio; a highly scalable technology; prominent reference enterprise customers, soon two of them (Virginia Tech and Colsa) due to climb to the top five of the supercomputer list; and its own big motivation ? for a company so often declared dead, grid computing could spell survival.
Until recently, running a cluster ? let alone a supercomputer ? stood within the reach of only a few wealthy and renowned institutions. While the hardware barrier remains a matter of fact, the soon-to-come ?gridability? of OS X will contribute to a more efficient use of existing network resources. Xgrid has all it takes to transform those idle Macs to your competitive advantage.
1) Ad-hoc parallel computing cluster:
You can combine any OS X-savvy Macs (including iBooks, iMacs, PowerMacs) using 100-MBit, Gigabit or Airport networks on the fly to create a so called ad-hoc parallel computing cluster.
All you need is at least two, preferably several Macs, a 100MBit Ethernet cable per Mac and switches with enough places to host all these connected cables. Switches for four to eight connections tend to be very affordable. 100Mbit switches for about 16-32 connections are more expensive and Gigabit switches are normally only used in dedicated parallel computing or other high-end network environments. The biggest advantage of such an ad-hoc parallel computing cluster is that you can even tap into an existing bigger network (Pooch, for example) over the Internet assuming that there is no firewall preventing you from accessing it. This always impresses an audience.
2) Student labs based on Macs
During the day, you have 25, 50 or maybe 100 Macs as single desktop machines hosting a computer lab. This environment often exists in universities and other higher education environments. When the students go home the Macs are transformed into just one single parallel computing cluster achieving up to hundreds of Gigaflops performance depending on the exact number of available machines.
This concept doesn't just sound appealing ? it has already happened at the University of Southern California (USC). Here, 76 dual-processor Power Macs are used together to achieve 233 Gigaflops of performance. This is an impressive achievement, given that the Macs are used in the daytime anyway and no additional hardware is required.
3) Dedicated Parallel Cluster
A dedicated cluster consisting of Power Mac G5s or Xserves is easily combined to deliver excellent and affordable performance. There are many examples: one cluster of 33 dual-processor XServe G5s achieved 217 Gigaflops at NASA's Jet Propulsion Laboratory. A very famous example is the dedicated Virginia Tech cluster, which consisted of 1,100 Power Mac G5s and achieved over 10 TeraFlops (1 TeraFlop is 1,024 Gigaflops). This cluster placed itself third fastest of the Top 500 Supercomputer list in autumn 2003. Now it's being upgraded to reappear again this autumn, this time based on G5 Xserves.
The impressive success of the Virginia Tech cluster has urged many to emulate the project, impressed by its unmatched ratio of performance to price. One of the first to tap into this new supercomputing opportunity at prices unseen before is COLSA, which aims to achieve performance of about 25 TeraFlops deploying a cluster of Xserves. Wherever large NT4 or Windows 2000 cluster have not been updated, because of Microsoft's high license fees and where Linux clusters are not seen as viable because of the cost of owning such set-ups, a dedicated Xserve cluster has emerged as a viable ? and often preferable ? option.
No matter whether the actually achievable Giga- and TeraFlop per price ratio of deploying Xserves is important, or the Storage Area Network (SAN) aspect of the Xserve RAID and the Giga- or TeraByte per price ratio is relevant, Apple's Xserve G5 and Xserve RAID are truly the best products available on the market.
4) Mixed Cluster
Finally, it's possible to combine any of these deployment scenarios. Very often you'll find a mix of permanent dedicated clusters that anyone can tap into as needed and additional computing performance that comes online when personal desktop workstations (PowerMacs or PowerBooks, for example) aren't being used.
Such normally 'wasted' computing power can be made available to the cluster on the fly, because any recent professional Mac ships with state-of-the-art Gigabit Ethernet connectivity ? even simple entry-level Macs offer at least a 100MBit Ethernet connection.