Remembering David E. Rumelhart (1942-2011)

APS Fellow and Charter Member David Everett Rumelhart, Professor of Psychology (Emeritus) died at age 68 on March 13, 2011 from complications of a progressive neurodegenerative disease.

After completing a PhD in mathematical psychology at Stanford in 1967, Rumelhart joined the faculty at University of California, San Diego, where he collaborated with others to lay out a computational theory of mind within the symbolic computational framework that was popular at that time. Then, in the mid-1970s, he became dissatisfied with the symbolic paradigm and turned his attention to exploring the potential for neural networks to capture the full flexibility and context sensitivity of human intelligence

Several major contributions flowed from this effort, including an interactive model of reading, a model of learning linguistic regularities without explicit rules, and a powerful algorithm for learning internal representations in multi-layered neural networks. Many of these contributions became central elements of a two-volume work entitled Parallel-Distributed Processing: Explorations in the Microstructure of Cognition by Rumelhart, McClelland, and the Parallel Distributed Processing Research Group (1986). Published at the crest of a new wave of interest in neural networks, this work had a huge impact on research in cognitive science and machine learning.

Rumelhart returned to Stanford in 1987, where he led a vibrant team of graduate students and post doctoral fellows until a neurodegenerative condition robbed him of his formidable intellectual powers in the 1990s. He retired to Ann Arbor, Michigan in 1998.

Rumelhart’s contributions were recognized by a wide range of honors and awards, including the MacArthur Genius Award, election to the National Academy of Sciences, and the American Psychological Association’s Distinguished Scientific Contribution Award. In the year 2000, a major international prize was established in his honor, the David E. Rumelhart Prize for Contributions to the Theoretical Foundations of Cognitive Science.

Throughout the active period of his career, Rumelhart was remarkable for his intensity, not only in science but also in sports like ping-pong and volleyball. As the remembrances below attest, many people continue to be inspired by his example of what can be accomplished in life — even one cut tragically short — by never holding back.

James L. McClelland
Stanford University

David Rumelhart was a member of the faculty, recently tenured, when I arrived at University of California at San Diego (UCSD) in the fall of 1974.  David had just produced an exciting new book called Explorations in Cognition with Don Norman and other members of the LNR Research Group (named for Peter Lindsey, who had left, and Don and David). The book described fun and interesting explorations applying the symbolic computational paradigm to the study of human memory and language.  I read it  with great interest and a bit of envy for those who had a chance to participate.

During my first year at UCSD I team-taught a triple course for honors psychology majors with David. The course had three nominally separate elements (content lecture course on cognitive psychology, methods in cognitive psychology, plus a laboratory), but we rolled it largely into one, and twice a week we had what essentially amounted to three-hour lecture blocks. I was terrified about teaching technically complicated material I hadn’t previously studied alongside a senior colleague who happened to be the author of the brand-new textbook we were using for the class. The book was great, and I suppose my lectures were alright, but I’ll never forget David’s lectures. There were two kinds. One kind involved David wandering into class with hardly any preparation and delivering what were sometimes exceedingly slow-paced and mumbled three-hour monologues. The other kind was completely different — animated, inspired, confident, and mind-expanding. I gained most of my grounding in computational perspectives on cognition from David’s book and from these special, inspired lectures.

A year or two later, I took David’s mathematical psychology course. His degree from Stanford was in mathematical psychology, and he approached the course as an opportunity to teach us how to go about creating mathematical models. There was no textbook, and we didn’t read the literature on existing models. Instead, we came to class with data. Each week, one of the students presented some data he or she had collected. David asked the student to offer an interpretation of the pattern, and usually the student had some tentative, but underdeveloped, thoughts. David would use the Socratic method to lead the student to some ideas that could be characterized in mathematical terms. He would then start writing equations on the board, reducing complex expressions to simpler ones, and teaching us the tricks of the trade by example. By the end of the session, we often had a complete mathematical characterization that accounted pretty well for the data. The most important thing about it for me was it showed me there was nothing especially mysterious about developing one’s own theoretical ideas. One just laid them out, put them into equations, and then worked out the implications.

I had some theoretical ideas of my own, and I had been inspired and empowered by the math psych class to try to develop them, but I was initially shy to share them with others. So I worked away alone on my first model, called the cascade model.  I did have bold goals for the work, wanting to provide an alternative to the ‘discete stage’ theory of mental processes that was prevalent in cognitive psychology.  Word got out that I was working on a theoretical paper though, so David sought me out to learn what I was up to. It was an amazingly helpful conversation. Although I had some great ideas, I hadn’t really worked out exactly how to explain why they were important. Using that same Socratic method, David sat with me in the same conference room where the mathematical psychology class was held and helped me to lay out the implications. The result was my first (and only solo) paper in Psychological Review.

Shortly after this, David and I began to collaborate on the interactive activation model of letter perception.  I had a quarter-long sabbatical during this period, adjacent to a summer, and we spent the better part of six months just working together developing the model, albeit with plenty of timeouts for long coffee breaks and trips to the tennis court. The initial conception behind the model came essentially from David (indeed it can be traced to his earlier paper on ‘An interactive model of reading’), and it was he who insisted we keep trying when it began to seem that the model might not account for one of the key phenomena in the literature. It was David who always steered us to keep things as simple as possible, and to leave aside thorny issues to focus on the essence of the issue. Keep it simple and stay focused: These were the characteristics of a successful cognitive modeler that I learned from David Rumelhart.

David seemed very shy until one got to know him; but in the course of our joint activities, we became very close.  Once he opened up, one felt a part of a special world with a special set of themes and goals. I remember feeling during this period that David was that older brother I’d always wished I’d had.

After the successful completion of the interactive activation model, David went off to Stanford for a year’s sabbatical, and during that time we both took up the study of what were then called “neural models” of perception and cognition. When David returned to UCSD, it seemed almost inevitable that we would rekindle our collaboration to develop a new grand exploration, one that turned into Parallel Distributed Processing: Explorations in the Microstructure of Cognition, by Rumelhart, McClelland, and the PDP Research group. After an intense six-month period in which David and I, together with Geoff Hinton and Paul Smolensky, laid out the overall framework for the PDP books, I went off to Boston for a sabbatical that had been planned long before, and Geoff Hinton moved on to Carnegie Mellon and a focus on Boltzmann machines. Even at a distance, though, David and I managed to play our separate roles in the development of the PDP volumes. During this time, I made steady progress applying ideas we had developed together to topics in memory and speech perception, and prepared my introductory chapter for the book.  David’s productivity seemed frustratingly meager through much of this period. When I asked him what he’d been up to he would say things like “oh nothing” or “working on a graphics program.” But in fact, he was developing the back-propagation learning algorithm and an application of Hopfield’s ideas about the computations that are performed by neural networks as the basis for a new way of thinking about cognitive structures (or “schemata” as he called them). The graphics program turned out to allow explicit visualization of what David called the Goodness Landscape, turning Hopfield’s Energy landscape upside-down. Both of these projects ended up being seminal contributions to the PDP volumes. We also had one more successful collaboration when we created our controversial model for learning the past tenses of English words. Although we spent only a few days together working out the key features of the model, it was a wonderful chance to work with my “older brother” again.

Stephen E. Palmer
University of California, Berkeley

I was a graduate student at University of California, San Diego who worked with Dave Rumelhart and Don Norman in the early 1970s. Just when I arrived in their lab, they began an ambitious, interdisciplinary research project on human memory and language understanding that formed the core of their research for that decade. Today, it would be called cognitive science, but when we were working on it, there was not, as yet, any such thing. In retrospect, it is easy to see that this project, which resulted in the publication of several influential papers as well as a unique book, Explorations in Cognition, by Norman, Rumelhart, and the LNR Research Group (1975), was one of the first major research efforts in modern cognitive science. In a marriage virtually unheard of at the time, the project successfully integrated psychological research on perception, attention, memory and language with linguistic theory and computer simulation techniques to explain the acquisition, storage, and use of knowledge from stories and other complex inputs. Dave was inspirational in the development of this model, later known as Elinor (for LNR: Lindsay, Norman, and Rumelhart, the name of our research group at the time). As was typical of Dave, he thought deeply about the big underlying questions, came up with elegant, creative answers, and shared his time and ideas freely with his colleagues. Perhaps my most inspirational memories of Dave came from his leadership of the “Verb Group,” a subset of the LNR lab members who wanted to understand how to represent the internal representation of verb meanings in the graph-structural formalism of the Elinor model. Many of the most important theoretical proposals were his own, but he made it easy for us to contribute our own ideas. My image is of him wandering into my office with a coffee cup in his hand and a boyish smile on his face, stroking his goatee, and saying, “You know, I’ve been thinking…” Dave was the quietest, most unassuming powerhouse of intellect I have ever met.

Donald Norman
Nielsen Norman Group

Few people play a major role in a scientific revolution. Fewer still are those who are fundamental to several revolutions. Dave Rumelhart qualifies.

Dave was a powerful intellect. He had significant influence upon many lives. For me, he was fundamental: He was a strong, wonderful colleague. Together we published 16 papers and one book.

I remember when we hired him at the brand-new department of psychology at the University of California, San Diego. He was extremely quiet and reserved, but he had a powerful intellectual presence. Peter Lindsay and I invited him to join our shared laboratory, and our band of three became the LNR Research Group, aided and abetted by an incredibly lively, talented group of graduate students. We were on a mission to change the world.

From the left: Donald Norman, David Rumelhart, Mark Wallen, and James L. McClelland, in a picture taken in the late 1970s.

Dave started as a mathematical psychologist, with superb math skills and great psychological insight. He and Peter played a critical role in the semantic-networks revolution of the study of human memory and reasoning. All three of us published many papers under various permutations of our names, including Explorations in Cognition, in which the LNR Research Group was listed as a co-author. Dave’s introduction of story schemas impacted research on story understanding and even research in developmental psychology. And then, of course, he helped bring about modern connectionism, also known as Parallel Distributed Processing (the title of the influential book series he co-edited) and neural networks. His influential work on motor skills and typing brought it all together: He combined psychological insight, mathematical modeling, and neural networks with an elegant computer simulation that showed fingers of moving across the keyboard, with some fingers moving to position letters ahead of the time the letters would be needed. The simulation did a wonderful job of matching the experimental data and high-speed motion pictures that the LNR lab had collected, but it was also a lot of fun just to watch. As Adele Abrahamsen, one of his early graduate students, put it in an email to me, “What was distinctive of Dave was how he constructed mathematical and computer models that he successfully applied to a whole range of human capacities, from perception to discourse.”

Dave was a towering intellect, a gracious colleague, and fundamental to the lives of his students and to the foundations of the modern psychological science.

Robert J. Glushko
University of California, Berkeley

Most of you know David Rumelhart’s foundational work as a mathematical psychologist and cognitive scientist, and perhaps you are familiar with the Rumelhart Prize (rumelhartprize.org), a $100,000 award given each year at the meeting of the Cognitive Science Society. The prize honors Dave’s scientific contributions each time another distinguished scientist receives it, but I believe that the Rumelhart Prize is unique among scientific prizes in an essential respect. When we created the prize, we made it an important requirement that the recipients must not only be distinguished scientists; they must also be people that we can look to as role models in the way they do science. This way, the prize honors Dave’s personal and professional generosity along with his scientific achievements.

I became Dave’s PhD student at University of California, San Diego (UCSD) in 1976, joining the joint research group that Dave, Don Norman, and Jay McClelland (a young assistant professor at the time) were running. There was an exciting intellectual energy in that lab at UCSD, and together they executed a coordinated attack on a very broad set of problems: representation of knowledge in memory, analogical learning, story grammar, speech perception, contextual effects in letter and word recognition, and interactive models of reading. But even with the remarkable breadth of work going on, this era was typical of Rumelhart’s scientific career. He always had a way of making very different things seem interconnected, and he never met a scientific problem he didn’t like. He had a voracious appetite for learning things (we thought he never slept), not in a competitive “my work is better than yours” way, but in a “let’s move together into the scientific jungle with our machetes and see if we can make a useful trail” way.

I remember Dave saying that you could always learn something from any talk or paper, even if they were not very good, because even then you were learning that “such and such” wasn’t a productive assumption or experimental approach. He didn’t have time to waste dwelling on anything that wouldn’t advance science. Think about how you react when you hear an incoherent presentation or read a shallow paper. We would all do better to follow Dave’s example.

This kind of personal and professional generosity defined the Dave Rumelhart I remember, and I could relate many similar stories, but I will just mention two more. Several of us were doing word recognition experiments of one sort or another, and we were choosing words as well as making up nonwords that were critical to the experimental control. We had to consider criteria such as frequency, word length, and orthographic structure, but this was an extremely tedious process and difficult to do well by hand. One day around 1977, Dave announced that he’d gotten a copy of the Kucera and Francis corpus of two million English words on magnetic tape, which we all knew would be extremely useful if we could put it to use. It seemed like just the next day Dave announced that he’d installed the corpus as a resource on our PDP-11 computer, so we only needed to write simple shell scripts to count and sort words. It made my dissertation research on neighborhood effects possible.

This was typical of the things Dave did to advance the quality of his science and the science of people around him. It might have been a little thing on its own to install the Brown corpus, but lots of little things like this have a substantial, cumulative impact. Dave seemed to write more code than his graduate students did, and he especially wrote a lot of tools and modeling and scaffolding code to make everyone do better science. How many of your thesis advisors wrote code that was essential to your thesis?

When I finished my PhD in 1979, most of the academic positions I looked into seemed too narrow because I’d been spoiled by the unique multidisciplinary collaboration environment that we had at UCSD. The one job I really wanted was one that sat in the intersection of linguistics, education, and cognitive psychology… but I didn’t get the job. I was quite disappointed and started looking into postdoc positions that didn’t appeal to me much, but I assumed that a postdoc was a necessary step to the academic career I thought I was inevitably headed for.  Dave told me that it was far more important that I went someplace that interested and valued me, and with that encouragement I opted out of an academic career and joined Bell Labs in an applied research area. A few years later I found myself in Silicon Valley doing “applied cognitive science” as a consultant and a co-founder of startups doing electronic publishing and Internet commerce. I have been successful enough in my endeavors, that I have been able to give back to the field where my professional roots are by creating and funding the Rumelhart Prize. Today, I’ve gone full circle and have an academic position at Berkeley, where I strive to follow the example that Dave set for me as a scientist, colleague and mentor.

Robert A. Bjork
University of California, Los Angeles

David Rumelhart arrived at Stanford University and the Institute for Mathematical Studies in the Social Sciences a year after I did. As two Midwestern types, David from South Dakota and me from Minnesota, we felt a kind of affiliation, and on arriving in Palo Alto we both realized that we weren’t in Kansas anymore, so to speak. David adapted to California in certain ways, such as buying a used orange Austin Healy convertible that he drove for a long time, but he was slow to adapt in other ways. It took years and being shaped by his wife, Marilyn, for example, to convince him that one could — and should — eat foods other than meat and potatoes.

David had been an academic star in South Dakota, both in high school and at the University of South Dakota. There is actually an entry in the Congressional Record about David’s undergraduate career, placed there by a South Dakota senator. That entry, though, is not so much about David as a model student, but about David as a model of how inexpensively one could attend the University of South Dakota. David lived in a church basement, earned his food by doing custodial work, and it cost him something like $50 a year to attend the University. Douglas Medin, the current president of APS, also spent a graduate year living in that church basement, so it may be the most distinguished basement in our field.

What David and I found at Stanford and Ventura Hall, where Dick Atkinson, Ed Crothers, Bill Estes, Patrick Suppes, and most of the graduate students in the mathematical psychology program were housed, was very exciting. We all thought we were at the center of the mathematical-modeling universe and discovering truth every day. Gordon Bower, though not housed in Ventura Hall, loomed large in our research and intellectual lives. Arguments went on long into the night and sometimes through the night in the Ventura Hall lounge, in part because if you took your deck of IBM cards over to the computer center late at night, you could get a rapid turnaround. David was in his element in those arguments, most of which had to do with how best to solve some problem or make something work.

Rumelhart was inducted into the National Academy of Sciences in 1991.

As graduate students, David and I worked closely together, especially when David got intrigued by some of the issues that motivated my dissertation research. We worked together on a daily basis, often several hours a day, and our dissertations ended up being essentially Parts One and Two of the same project. We also pursued other important challenges together, such as playing ping-pong to exhaustion (which will come as no surprise to David’s students) and learning to throw a boomerang  — not a toy boomerang, but a large genuine one, which proved to be very frustrating. For every throw that achieved perfection, there were 20 that either went too high and then accelerated back down with lethal intent, or went too low, caught a edge on the ground and went end over end away from us for a 100 yards or so. Only David’s tenacity kept us returning again to Stanford’s athletic fields in an effort to solve the puzzle of boomeranging. We could have been filmed to illustrate the amazing resistance to extinction that is created by a variable-ratio schedule of reinforcement.

What was indeed unique about David was his tenacity — and pure enjoyment — at trying to figure things out and/or make something work. When we would start to work on something, he would be all hyped up, like an actor about to go on stage or an athlete about to jump at center court or field a kickoff. The harder he worked, and the more things seemed puzzling, the more animated and excited he got. He didn’t like to wait for answers either. When the programmers at the Institute of Mathematical Studies said that it would be a couple weeks before they could complete the programming necessary to see whether a particular Markov model would fit my dissertation data, David and I, instead of waiting, learned Algol and did the programming ourselves. In my case, a major motivation was being able to finish my dissertation before I had to leave for a faculty position at Michigan. In David’s case, the motivation was simple: He wanted to know the answer. When something puzzled David, he was like a dog with a bone and he wouldn’t stop chewing on it. There may be no better example than when he figured out how back propagation might tune connectionist models, one of the major contributions of his career.

David was truly unique. Working together, we developed a kind of private language, triggered, as I recall, by taking a logic course with Patrick Suppes. I have forgotten the vocabulary of that language, except for one term, “gar-bar,” which was David’s ultimate complement with respect to some idea or model. Gar referred to garbage, so gar-bar meant not garbage. David, as a scientist, colleague, and mentor, you were the ultimate gar-bar.

Andreas S. Weigend
Social Data Lab

Dave Rumelhart was giving me a ride from the Santa Fe Institute to Los Alamos when I finally grasped the true meaning of a random variable. It was in the summer of 1990, four years into my PhD at Stanford University, and Dave had offered to drive. Only then did I realize how deep an understanding of basic concepts can be. As the sun set over the desert horizon, Dave said, “Andreas, I want you to understand what it really means.” Ever since that car ride, I have come to thank Dave for his willingness to share his intuitions about simple, yet powerful, concepts.

I had come to Stanford from Germany in 1986 to pursue a doctorate in physics. Yet after reading Chapter 8 of Parallel Distributed Processing (PDP), I became very interested in the possibilities of learning from data and then, in my second year at Stanford, I asked Dave to become my PhD advisor. Always open to new ideas, he agreed, and I joined his lab, known as the PDP Research Group. Dave was known to accept people from many different backgrounds and disciplines. However, we hardly ever talked about the “interdisciplinary” nature of our endeavors — it was just part of the fabric of the group. Dave believed in integrating ideas and tools irrespective of their origin.

Soon, it became apparent to me that there were actually two Dave Rumelharts. The first was the brilliant professor who showed up every morning in my office, seeing things nobody else did: what direction to go in, what to try, and how results might change. He had an amazing mental model based on his natural acuity and his vast experience running simulations. Yet… it seemed to us that there was a second Dave Rumelhart, a mysterious full-time world-class programmer who, when one of us came up with an idea, would whip up a piece of code overnight and send over an email that read, “All done. Try it out!”

In the end, the quality that inspired me most was Dave’s passionate and profound ability to communicate and to co-create. He did whatever it took, including using his own sock as a whiteboard eraser to create a blank slate and “take another tack at it”, as he would say. The German word for Ph.D. advisor is Doktorvater, doctor-father. Dave truly was a mentor and father to me.