New scheduling for ‘Thinking about the Hardware of thinking’

I was scheduled to give a live virtual seminar, streamed to the Transvision conference in Italy on October 23rd. Unfortunately I was not able to deliver the presentation due to technical problems at the conference venue.

But the good news is, I will be giving the talk this weekend instead!

Here is the abstract (slightly updated as the talk will be a little longer than originally planned)

Thinking about the hardware of thinking:
Can disruptive technologies help us achieve uploading?

S. Gildert,
Teleplace, 28th November 2010
10am PST (1pm EST, 6pm UK, 7pm continental EU).

We are surrounded by devices that rely on general purpose silicon processors, which are mostly very similar in terms of their design. But is this the only possibility? As we begin to run larger and more brain-like emulations, will our current methods of simulating neural networks be enough, even in principle? Why does the brain, with 100 billion neurons, consume less than 30W of power, whilst our attempts to simulate tens of thousands of neurons (for example in the blue brain project) consumes tens of KW? As we wish to run computations faster and more efficiently, we might we need to consider if the design of the hardware that we all take for granted is optimal. In this presentation I will discuss the recent return to a focus upon co-design – that is, designing specialized software algorithms running on specialized hardware, and how this approach may help us create much more powerful applications in the future. As an example, I will discuss some possible ways of running AI algorithms on novel forms of computer hardware, such as superconducting quantum computing processors. These behave entirely differently to our current silicon chips, and help to emphasize just how important disruptive technologies may be to our attempts to build intelligent machines.

Here is a link to the Teleplace announcement.

Hope to see you there!

Building more intelligent machines: Can ‘co-design’ help?

Here is a little essay I wrote in response to an article on HPCWire about hardware-software co-design and how it relates to D-Wave’s processors. I’ve also put this essay in the Resources section as a permanent link.

Building more intelligent machines: Can ‘co-design’ help?

S. Gildert November 2010

There are many challenges that we face as we consider the future of computer architectures, and as the type of problem that people require such architectures to solve changes in scale and complexity. A recent article written for HPCwire [1] on ‘co-design’ highlights some of these issues, and demonstrates that the High Performance Computing community is very interested in new visions of breakthrough system architectures. Simply scaling up the number of cores of current technologies seems to be getting more difficult, more expensive, and more energy-hungry. One might imagine that in the face of such diminishing returns, there could be innovations in architectures that are vastly different from anything currently in existence. It seems clear that people are becoming more open to the idea that something revolutionary in this area may be required to make the leap to ‘exascale’ machines and beyond. The desire for larger and more powerful machines is driving people to try to create more ‘clever’ ways of solving problems (algorithmic and software development), rather than just increasing the speed and sheer number of transistors doing the processing. Co-design is one example of a buzzword that is sneakily spreading these memes which hint at ‘clever’ computing into the HPC community.

Generalization and specialization

I will explain the idea of co-design by using a colorful biological analogy. Imagine trying to design a general purpose animal: Our beast can fly, run, swim, dig tunnels and climb trees. It can survive in many different environments. However, anyone trying to design such an animal would soon discover that the large wings prevented it from digging tunnels effectively; that the thick fur coat to survive the extreme cold was not helpful in achieving a streamlined, fast swimmer. Any animal that was even slightly more specialized in one of these areas would quickly out-compete our general design. Indeed, for this very reason, natural selection causes specialization and therefore great diversity amongst the species that we see around us. Particular species are very good at surviving in particular environments.

How does this tie in with computer processing?

The problems that processors are designed to solve today are mostly all very similar. One can view this as being a bit like the ‘environmental landscape’ that our general purpose creatures live in. If the problems that they encounter around their environment on a day-to-day basis are of the same type, then there is no reason to diversify. Similarly, a large proportion of all computing resources today address some very similar problems, which can be solved quite well using general purpose architectures such as Intel Centrino chips. These tasks include the calculations that underlie familiar everyday tasks such as word-processing, and displaying web pages. But there do exist problems that have been previously thought to be very difficult for computers to solve, problems which seem out of reach of conventional computing. Examples of such problems are face-recognition, realistic speech synthesis, the discovery of patterns in large amounts of genetic data, and the extraction of ‘meaning’ from poetry or prose. These problems are like the trees and cliffs and oceans of our evolutionary landscape. The general purpose animals simply cannot exploit these features, they cannot solve these problems, so the problems are typically ignored or deemed ‘too hard’ for current computing platforms.

But there are companies and industries that do care about these problems. They require computing power to be harnessed for some very specific tasks. A few examples include extracting information from genetic data in the biotechnology companies, improving patient diagnosis and medical knowledge of expert systems in the healthcare sector, improving computer graphics for gaming experiences in entertainment businesses, and developing intelligent military tools for the defense industry. These fields all require the searching and sorting of data in parallel, and the manipulation of data on a much more abstract level for it to be efficient and worthwhile. This parallel operation and abstraction is something that general purpose processors are not very good at. They can attempt such a feat, but it takes the power of a supercomputer-size machine to tackle even very small instances of these specialized problems, using speed and brute force to overwhelm the difficulty. The result is very expensive, very inefficient, and does not scale well to larger problems of the same type.

It is this incorporation of variety and structure, the addition of trees, cliffs and oceans, into our computational problems causes our general-purpose processors to be very inefficient at these tasks. So why not allow the processors to specialize and diversify, just like natural selection explores the problem environment defined by our biological needs?

Following nature’s example

Co-design attempts to address this problem. It tries to design solutions around the structure of the problem type, resulting in an ability to solve that one problem very well indeed. In practice this is done by meticulous crafting of both software and hardware in synchrony. This allows software which complements the hardware and utilizes subtleties in the construction of the processor to help speed things up, rather than software which runs on a general architecture and incurs a much larger overhead. The result is a blindingly fast and efficient special purpose architecture and algorithm that is extremely good at tackling a particular problem. Though the resulting processor may not be very good at certain tasks we take for granted using general-purpose processors, solving specialized problems instead can be just as valuable, and perhaps will be even more valuable in the future.

A selection of processors which are starting to specialize are discussed in the HPCwire article. These include MDGRAPE-3, which calculates inter-atomic forces, and Anton, a system specifically designed to model the behaviour of molecules and proteins. More common names in the processor world are also beginning to explore possible specializations. Nvidia’s GPU based architectures are gaining in popularity, and FPGA and ASIC alternatives are now often considered for inclusion in HPC systems, such as some of Xilinx’s products. As better software and more special purpose algorithms are written to exploit these new architectures, they become cheaper and smaller than the brute-force general purpose alternatives. The size of the market for these products increases accordingly.

The quantum processors built by D-Wave Systems [2] are a perfect example of specialized animals, and give an insightful look into some of the ideas behind co-design. The D-Wave machines don’t look much like regular computers. They require complex refrigeration equipment and magnetic shielding. They use superconducting electronics rather than semiconducting transistors. They are, at first inspection, very unusual indeed. But they are carefully designed and built in a way that allows an intimate match between the hardware and the software algorithm that they run. As such they are very specialized, but this property allows them to tackle very well a particular class of problems known as discrete optimization problems,. This class of problems may appear highly mathematical, but looks can be deceiving. It turns out that once you start looking, examples of these problems are found in many interesting areas of industry and research. Most importantly, optimization forms the basis of many of the problems mentioned earlier, such as pattern recognition, machine learning, and meaning analysis. These are exactly the problems which are deemed ‘too hard’ for most computer processors, and yet could be of incredible market value. In short, there are many, many trees, cliffs and oceans in our problem landscape, and a wealth of opportunity for specialized processors to exploit this wonderful evolutionary niche!

Co-design is an important ideas in computing, and hopefully it will open people’s minds to the potential of new types of architecture that they may never have imagined before. I believe it will grow ever more important in the future, as we expect a larger and more complex variety of problems to be solved by our machines. The first time one sees footage of a tropical rainforest, one can but stare in awe at the wonders of never-before-seen species, each perfectly engineered to efficiently solve a particular biological problem. I hope that in the future, we will open our eyes to the possibility of an eco-sphere of computer architectures, populated by similarly diverse, beautiful and unusual creatures.



Transvision2010 presentation: Thinking about the hardware of thinking

I will be giving a presentation at Transvision2010, which takes place next weekend. The talk will be about how we should consider novel computing substrates on which to develop AI and ASIM (advanced substrate independent minds) technologies, rather than relying on conventional silicon processors. My main example will be that of developing learning applications on Quantum Computing processors (not entirely unpredictable!), but the method is generalisable to other technologies such as GPUs, biologically based computer architectures, etc…


The conference is physically located in Italy, but I unfortunately cannot make in in person, as I will be attending another workshop. I will therefore be giving the talk remotely via the teleconferencing software Teleplace.

Anyway, here is some information about the talk, kindly posted by Giulio Prisco:

Thinking about the hardware of thinking:
Can disruptive technologies help us achieve uploading?

ASIM-2010 – not quite Singularity but close :)

So I’ll post something about the Singularity Summit soon, but first I just wanted to talk a little about the ASIM-2010 conference that I helped organise along with Randal Koene.

The main idea of the conference was to hold a satellite workshop to the Singularity Summit, with the purpose of sparking discussion around the topics of Substrate Independent Minds. See the carboncopies website for more information on that! Ah, I love the format of blogging. I’m explaining what happened at a workshop without having introduced the idea of what the workshop was trying to achieve or what our new organisation actually *is*. Well, I promise that I’ll get round to explaining it soon, but until then it will have to be a shadowy unknown. The carboncopies website is also in the process of being filled with content, so I apologise if it is a little skeletal at the moment!

One interesting thing that we tried to do with the workshop was to combine a real life and a virtual space component. It was an interesting experience trying to bring together VR and IRL. In a way it was very fitting for a workshop based around the idea of substrate independent minds. Here we were somewhat along the way to substrate independent speakers! I am hoping that this will inspire more people to run workshops in this way, which will force the technology to improve.

I was very pleased too see so many people turning out. We had about 30 people in meatspace and about another 15 in virtual space on both days. Giulio Prisco has some nice write-up material about the workshops, including PICTURES and VIDEOS! Here are the links to his posts:

General overview
First day in detail
Second day in detail

For a first attempt, I don’t think that things went too badly! The technology wasn’t perfect, but we gave it a good try. The main problem was with the audio. Teleplace, the conferencing software we were using, works well when everyone in online with a headset and mic, there are no feedback problems. However, when you try and include an entire room as one attendee, it becomes a little more tricky.

This could be improved by either everyone in the room having headsets and mics, and then having a mixer which incorporated all the input into a single Teleplace attendee. The other way is that everyone in the room could also be logged into Teleplace with their own headsets and mics. *Make* that Singularity happen, ooh yeah! (/sarcasm)

A particularly bad attack on the Singularity

Whilst I am not a raging advocate of ‘sitting back and ‘waiting’ for the Singularity to happen (I prefer to get excited about the technologies that underlie the concept of it), I feel that I have a responsibility to defend the poor meme in the case where an argument against is is actually very wrong, such as in this article from Science Not Fiction:

Genomics Has Bad News For The Singularity

The basic argument that the article puts forward is that the cost of sequencing the human genome has fallen following a super-exponential trend over the past 10 years. And yet, we do not have amazing breakthroughs in drug treatment and designer therapies. So how could we expect to have “genuine artificial intelligence, self-replicating nanorobots, and human-machine hybrids” even though Moore’s law is ensuring that the cost of processing power is falling? And it is falling at a much slower rate than genome sequencing costs!

The article states:

“In less than a decade, genomics has shown that improvements in the cost and speed of a technology do not guarantee real-world applications or immediate paradigm shifts in how we live our day-to-day lives.”

I feel however, that the article is somewhat comparing apples and oranges. I have two arguments against the comparison:

The first is that sequencing the genome just gives us data. There’s no algorithmic component. We still have little idea of how most of the code is actually implemented in the making of an organism. We don’t have the protein algorithmics. It’s like having the source code for an AGI without a compiler. But we do have reasonable physical and algorithmic models for neurons (and even entire brains!), we just lack the computational power to simulate billions of neurons in a highly connected structure. We can simulate larger and larger neural networks as hardware increases in speed, connectivity, and efficiency. And given that the algorithm is ‘captured’ in the very structure of the neural net, the algorithm advances as the hardware improves. This is not the case in genome sequencing.

The second argument is that sequencing genomes is not a process that can be bootstrapped. The very process of knowing a genome sequence isn’t going to help us sequence genomes faster or help you engineer designer drugs. But building smart AI systems – or “genuine artificial intelligence” as the article states – CAN enable you to bootstrap the process, as you will have access to copyable capital for almost no cost: Intelligent machines which can be put to the task of designing more intelligent machines. If we can build AIs that pass a particular threshold in terms of being able to design improved algorithmic versions of themselves, why should this be limited by hardware requirements at all? Moore’s law really just gives us an upper bound on the resources necessary to build intelligent systems if we approach the problem using a brute-force method.

We still need people working on the algorithmic side of things in AI – just as we need people working on how genes are actually expressed and give rise to characteristics in organisms. But in the case of AI, we already have an existence proof for such an object – the human brain, and so even with no algorithmic advances, we should be able to build one in-silico. Applications for genomics do not have such a clearly defined goal based on something that exists naturally (though harnessing effetcs like the way in which cancer cells avoid apoptosis might be a good place to start).

I’d be interested in hearing people thoughts on this.

Essay: Language, learning, and social interaction – Insights and pitfalls on the road to AGI

This is the first in a series of essays that I’m going to make available through this site. I’m going to put them in the resources page as I write them.

I’m doing this for a couple of reasons. The first is that I like writing. The second is that a blog is not the best format to present ideas that you need to frequently refer to, and often when you get asked the same question by multiple people it is better to point them to a permanent resource rather than repeating yourself each time. The third is that I would like to add some ideas to the general thought-provoking mash-up of AGI memes around the internet. The fourth is that I think people enjoy reading short articles and opinion pieces somewhat more than entire books. The fifth is that (somewhat in contradiction to my previous reason) I’m hoping to eventually write a neat book about related topics, and whilst I have already started doing this, I feel that I need a lot more practice writing several-page documents before I feel I can make something that is 100+ pages long which I am happy with. Note, the PhD thesis does not count!! 😉

So here we go. Click on the title to download the PDF.
You can also find a copy of the essay in the resources tab.

Language, learning, and social interaction
Insights and pitfalls on the road to AGI

Why is language important? What purpose does it serve on a fundamental level? How is it related to the ways in which we learn? Why did it evolve? In this short essay I’m going to take a Darwinian look at language and why we must be careful when considering the role played by language in the building of intelligent machines which are not human-like.

H+ Summit 2010 @ Harvard

I’m currently in Harvard listening to some pretty awesome talks at the H+ Summit. I always really enjoying attending these events, the atmosphere is truly awesome. So far we have had talks about brain preservation, diy genomics, neural networks, robots on stage, AI, consciousness, synthetic biology, crowdsourcing scientific discovery, and lots lots more.

The talks are all being livestreamed, which is pretty cool too. I can’t really describe the conference in words, so here are some pictures from the conference so far:

Audience pic:

General overview:

Here is a picture of Geordie’s talk about D-Wave, quantum computing and Intelligence:

Here is a picture of me next to the Aiken-IBM Automatic Sequence Controlled Calculator Mark I. This thing is truly amazing, a real piece of computing history.

The MIT museum was also really cool 🙂 More news soon!