Maybe the past generation of programmers was sort of subjected to a really awful version of visual programming, but it would be really sad to let that frustration that other people have felt ruin what visual programming could be in a much better design format or just done in very different ways, done in ways that combine graphics and text.
Hello and welcome to Metamuse. Muse is a tool for thought on iPad. But this podcast isn't about Muse the product. It's about Muse the company and the small team behind it. I'm Adam Wiggins joined by my colleague, Mark McGranaghan,
Hey Adam.
And our guest Maggie Appleton from Egghead.
Hello.
Thanks for joining us. Maggie. Now listeners to the podcast will know that Mark and I are fans of computing history. We think sometimes the technology industry is a little unaware of its own history. And I understand you recently visited a museum on that topic in your area.
I did. I got last weekend to go up to Cambridge, which is about an hour away from London and visited the Computer History Museum of Cambridge, which was mostly a warehouse that some people had put an old N64 in and an old Mac and got to play a bit of Pong and Space Invaders, which is terribly educational, but probably not a comprehensive history of computing.
Now, depending on what generation you grew up with, you know, so now I'm old enough that I go to a computer history museum, or sometimes there's like the pinball museum in Alameda, there in San Francisco or some of these arcade ones. And I remember this stuff when it was new, maybe as a kid. So in that case, it can be as much a sense of nostalgia or a comparison with your sort of adult perspective on something that loomed large as a child. Were there things there you recognized or was this all like essentially stuff that predated your time?
I was definitely of the N64 generation and definitely the, I don't know which version of the Mac it was, but I have a very clear memory of going into a friend's house who had the Mac. Windows 95 was maybe what we had had at home. I mean, I was grew up sort of in the noughties. So I'm that generation.
Is that we're calling it now I've heard the naughts.
I know the naughts. There's no good word for it. I mean.
The zeroes. Turn of the century, maybe we could go with.
So before we get into our topic, love to hear a little bit about your background, Maggie, as well as a little bit about what you do at Egghead.
Sure. So yeah, Egghead is where I currently work and it's a web development education platform. So we kind of we'll joke it's Netflix for developers. It's mostly video tutorials, teaching anything mostly on JavaScript, lots of React, Angular, Vue, that sort of thing. And I'm the art director/designer/illustrator there. So essentially I handle most of anything that is visual design. And I also do quite a bit of UX design there. It's sort of a small company so you get to be a bit of an all-rounder and because it's a very developer focused company, I also develop mostly working in React. I like to call myself a mediocre developer, so I'm not intense, you know, I don't really like dig into, you know, the deep code, but I know my way around, you know, front end database. So yeah, that's kind of me.
It seems like you came maybe from the, I dunno, illustration or design side, and then that led you to kind of the development side or was it the other way around?
Yes, it was definitely design and illustration first. So I had grown up in a very technological household. My parents were both programmers. I had a lot of access to technology, quite young, so I learned, you know, HTML and CSS, but I wasn't really introduced to proper programming young and you know, there wasn't any like IT education in school. So I went more into design and illustration in my early twenties and then learned programming mostly out of employment necessity. You know, I was, I was designing websites and, you know, starting to learn JavaScript a bit and figuring out how to get a modal to work. And then of course, once I started working at Egghead, got a lot more into programming and development. So I very much come at it from, I struggled to learn it. I'm maybe not, I don't want to get into natural types of people, but of course some people have an affinity for being able to do abstract reasoning.
And they're very good programmers because they can hold a ton of cognitive context in their head about what component is connected to what database, you know, where messages being passed. And I am not strong on a lot of those fronts. So I do development as someone who needs things visually displayed in front of me. I need a lot of explicit feedback from the machine telling me what's happening and what things are connected. So my perspective I kind of bring to programming is I'm the person that struggles with it. So I really, really want tools that actually help you program if you're not naturally inclined to abstract reasoning.
I think it's interesting that you saw basic programming through HTML and JavaScript as being, it almost sounds like a table stakes for your career. And I think that's the nature of the computing medium. You can imagine doing visual illustration design work in print, for example, and there, you probably need to learn some things about printers, but for the most part, you're fine just doing everything in pixels on the screen. You hand that over and there you go. And that's probably true in other realms as well, but once you're on the computer, now you're working in the dynamic medium and your ability to express things is very limited. If you're stuck with what's static, I think that's going to become more and more true here in the future.
Yeah, I think it was good. I was ignorant of the fact at the time that I was in the generation where people were designing websites in Photoshop and I didn't know that was an option. So I went and learned CSS and JavaScript because I couldn't figure out how any other way to build a website. I didn't know I could just mock up pixels and hand it to a developer. So I'm very glad I was ignorant of that.
So our topic today is visual programming. And also to some degree its relationship to the learnability of programming. And since you suggested this one, Maggie, maybe you can explain a little bit for the audience what the basics-- What is visual programming? Why is it interesting area?
Sure. So visual programming is sort of a catch-all name for a whole bunch of approaches to trying to do programming differently. I want to say, I think the name itself is problematic, which we can get into a bit, but it's essentially trying to explore graphical interfaces for programming in various ways. And there's been a long history of people trying to do this and a fair amount of pushback to versions of it that have gone quite badly, which we can also get into. But yes, it's essentially trying to explore ways of doing programming that is not just based in text, you know, linear text in a text editor, which is kind of the default nowadays.
Yeah. Certainly when people think of programming, they think of it as being synonymous with coding. I think of that Hollywood hacker view where they're sitting in front of the terminal, the text is probably green on black and it's a bunch of text in this computer language. Now, is there a well-known example of what would count as visual programming?
The two that jumped to my mind I'd say most famous is probably MIT has a programming language called Scratch that is technically designed for children in order to teach kids programming sort of as their first programming language. And I think this gets referenced a lot in visual programming circles, because it gets rid of needing to use sort of arbitrary syntax symbols to define things like objects and functions. Instead it uses drag and drop blocks with colors that correlate to the type of each element. So it's a very interesting example of how you can kind of lower the bar to people coming in and doing programming since it is designed for children. So that's one that gets famous, but others that I would count as being some form of visual programming that will be very familiar to the audience are tools like Zapier, If This Then That, Integromat. Now the sort of the whole low code, no code movement where people are essentially building node based logic flows, where they're saying, you know, if this happens, then do this else do that. They're doing, you know, programmatic logic, but they're doing it in a very graphical interface. And I count that as visual programming, but I think there's also a wide diversity of, we can talk about, you know, the amount of power each of these platforms might have the amount of flexibility and control they give to the user. But yeah, those would count, I would say as visual programming,
One it comes from my childhood, thinking of kids learning programming is, there's a game called Rocky's Boots. It was like a 1980s, ran on the Apple II or something. And it showed you how to put together logic gates, the same kind that you use in electrical engineering to make circuits and solve problems. And that was one of the things that maybe introduced me to this sort of formal logic that eventually led to my interest in programming. But one thing I think is interesting about that is in fact, electrical engineers, designing circuits work with a form of visual program. You make a diagram that shows where the resistors and the capacitors and the LEDs and whatever else go in this form that is very much, I guess it's less of a flow diagram and more shows where the electricity flows, which may or may not exactly represent sort of the abstract logic. But I think that's a professional context where there's kind of a visual programming at least representation.
And we can also talk about examples of visual augmentation. So with Scratch kind of the whole environment is inherently very visual. Other examples we could look at are spreadsheets where you have the cells themselves are like texts formulas more like a typical programming language, but it has this visual element of layout and spatial based variable naming. Another example from when I was learning programming is the, I think it was called Dr. Scheme. I think it has a new name now, or maybe a change to that from something else I forget, but it was the scheme development environment, which first of all was all in one, which is great. So you've got this like one executable, you could run and do everything, but also when you hit errors, which is the dreaded case of a new programmer, it just drew lines to everything you needed to look at, which sounds so basic, but just connecting the dots between this is your call site, and this is your function definition and there's something wrong, you know, check where these two lines terminate and see what's up. We can talk about all kinds of examples of visual augmentation As well.
Yeah. And I know you guys are also familiar with a few, maybe more powerful visual languages like Origami. And I think Xcode is quite visual in the way that it allows people to program. So yeah, they're all fully fledged IDEs that have visual elements to them and might not be fully fledged visual programming environments. But the hybrid is actually much more interesting than this idea of pure visual programming, but we can get there.
Our colleague Leonard uses Origami for some design prototyping. We even put some screenshots into one of the beta memos we did. I'll link that, but it's interesting because he's doing design work and typically he can do that in a tool like Figma and pretty static mock-ups. And then we go from there working with Julia, our interface engineers, they connect the static screens to behavior, but there are certain things. For example, when we were working on this whole infinite canvas flex boards feature where just how it felt and how it behaved was the whole design. And you just couldn't evaluate that by looking at a static screen. And furthermore, just the standard click prototypes, like you get in a lot of modern design tools just won't cut it. And so he ended up doing some fairly complex programs that you could then in turn, run on the iPad and really feel how it worked. It's quite impressive.
Yeah it's going to be very tempting to market this episode with a D&D style alignment chart for visual programming tools like lawful good is, you know, Scratch and what's the other corner chaotic evil is a meme generator is a programming tool or something like that. We'll see if we can pull it off.
Yeah. I'm thinking of other ones I've seen. I think there's a category of them that I think was very popular in the nineties. I was specifically thinking of LabVIEW gets a lot of negative press I'll say is maybe not chaotic evil, but it's one of the infamous ones that I think was proprietary. So you kind of got locked into the system and then there's a whole Tumblr. We can put it in the show notes when I find the link to it. But I can't remember the name now that is just screenshots of awful literal spaghetti code because it's a node-based visual programming language. So it would just be thousands of boxes and lines running between all of them and literal just a visual spaghetti that you can't tell down from up. But yeah, that would be somewhere in the evil dimension.
Yeah. The boxes connected by lines also reminds me of another big category that I've come across in my, I guess not my day job would be the way to put it, but music production is something that's been of great interest to me, particularly earlier in my life. And so I worked a lot with software, like Logic Audio, for example, sequencers or something like Reason, which is a kind of an all-in-one synthesizer plus sequencer. And these typically have various ways to essentially wire stuff together. But I think one reason that the visual thing works well there, is you have this clear kind of input/output flow. The output is always going to be an audio stream or in some cases there's video tools as well, where there's a video stream and it's a single sort of time-based flow. And so like a music track or something like that, you've just got seven minutes of music output and you can in Logic, for example, go to the quote unquote environment and there you can just string together. Okay. I want to take the source material. I don't know someone playing a guitar and I'm going to put it through these filters and I'm going to fork it off this way. And then the end output just goes out through, you know, your master volume. And obviously that is very much designed for professional, very serious use cases, but these are non-programmers.
The visual effects industry has a lot of those same sort of interfaces. So the node model box in there, I was just connecting different things together and really just showing data transformation over time, which is when a flow diagram actually comes in useful.
And then another one I think it'd be remiss not to mention is iOS has a automation system built in called Shortcuts, which was originally a very cool indie app that apple acquired called Workflow. And this essentially gives you sort of a Scratch style model, but probably even simpler in the sense that you don't have a lot of nesting. And what have you, you just sort of assemble these blocks top to bottom. You can do it on a phone, which actually is pretty interesting that it's possible to do reasonable programming on a phone. And again, I think it does rely on a fairly linear input and output. You're going to apply your shortcut to this image. You're going to do some stuff to it. Something comes out the other side. Maybe it's a transformed image. Maybe it's something else, but that's a good example of a place where visual programming maybe is actually the mainstream in a way that a lot of folks don't even acknowledge.
Yeah it kind of transitions us into the idea that visual programming ends up being this really large umbrella that I think a lot of the debates around to get into people, just not having an agreed definition of what we're really talking about. And so I mentioned we would come back to it being that it's kind of a terrible name because to call something visual programming suggests that it is something visible with human eyes, which would most certainly include text or a screen. So that would include all programming. So when people say visual programming, I think what it's really getting at is this is my sort of opinion on it I suppose. It's trying to take things that would otherwise be implicit in programming. So parts of the program we don't see, so right data transforming over time or a variable being passed through a function and making it visible in some way.
And this is kind of where it gets into the balance that visual programming platforms or languages have to kind of grapple with is how much do you make visible? And in what context, because you can't make everything visible because the point of programming is abstraction and working with higher level primitives and sort of taking away things we don't want to always deal with, right? Like where memory is allocated, you don't want that visible in a higher level programming language. So I always see it as how we take programming and we make different aspects of it visible in different contexts, depending on what we're trying to do.
Is there a better name, or if you were in charge of the rebrand on this somehow, what would you go for?
I feel like graphical programming would at least more accurately describe what a lot of these platforms are trying to do. I mean, a lot of them are simply just trying to apply better UX and interaction design to programming, right. They're just trying to make it a more enjoyable experience that gives the user or in this case, you know, it's a developer, what they need at the right time without overloading them with information, you know, it really just being a good cognitive tool for them to get their work done. So, yeah, but graphical programming, I think would more accurately describe node based interfaces or drag and drop, you know, ones like Scratch or even debuggers really, I think are a form of visual programming in a way, ones that really walk you through how the program is executing over time. Although I think it's kind of wild that you wait until something goes wrong until those tools come into the picture. Like why isn't there an inline debugger in every IDE, but you know.
Actually one of the first sort of, one of my mentors when I was getting started with the professional programming, he strongly felt that you would go and put break points at every interesting point in the code. The first time you ran it and you wouldn't run it and just see what happens, you're going to step through inspect the variables, watch every single point. And only once you've been through it all and watched it all execute from the inside. And everything's what you expect. Then you can look and see what lands on the screen, if that's what you expected or not, but that's kind of the final step. That's more your victory lap, not your first test of whether, what you did worked.
I think you mentioned that console logs, right. Are one method of sort of putting out or making the program more visible at certain points in time. I was trying to think of a good metaphor for what console logs are, right. It's like if the stream of the program, and it's like, you're shooting up a small bit of information at a point in time.
Yeah. It's sort of like if you have a map of a territory and you take a journey along the territory. We go in loops and you cross back over your own path and you make jumps and stuff kind of captures all that in a linearized form. And then I see as a compliment to the map, you know, the map gives you the nature of the-- Now people can get mad at me on the internet for saying that, but you know that the map has some correlation to the territory, hopefully, but also the linear view of everything that you encountered is also very useful.
One idea I would raise for the rebrand question I asked earlier was in the Ink and Switch piece we did on end-user computing. And we can talk about the relationship of visual programming to end user computing. But we talked about this property of embodiment and it's exactly what you said, the idea of taking something implicit and potentially very abstract and making it visible. And I think a lot of times in discussions I've seen about this, which are often among people who are already professional software engineers. And that includes both people who are very enthusiastic about end user programming, or just increasing the accessibility of programming, but they themselves are still professional programmers. I have that background, Mark does as well. And whether or not they're pro or con they're coming from that perspective of someone who, for whatever reason found it was drawn to, or was good at holding all of that abstraction in their minds.
And so in some ways, I wonder when I see the objection of, I don't know, there's so much visual spaghetti here, but that spaghetti all exists in symbolic code as well. You just can't see it and you have to mentally model it. And some people are unbelievably good at that. And a lot of those people are people who go on to be really great programmers. I'm probably kind of in the middle on that. My complexity threshold is not that great. And I tend to always want to like factor my code in a way to make it more comprehensible and give myself more visibility and that sort of thing. So I wonder how much the needs of visual programming, or this umbrella we're calling visual programming, those serve sort of hypothetical people that today are not programmers, but maybe would have that capability, which I think is the other dimension of this. Right. Which is the learnability side of it.
Yeah. I definitely agree with that, that I do like to be the representative, for, yeah, mediocre programmers or people who aren't naturally inclined to it and that I have heard a lot of the debates around it of people saying, well, I love Vim. Don't take Vim away from me. You know, if you try to make it visual, I won't be able to work in my crazy text editor with all my customized hotkeys. And I'll say like, textual programming is not going anywhere. You know, it's like, this is all going to be additional affordances built on top of a textual paradigm. You know, text is not leaving, but people say they can't understand why we would ever need it. Whereas my perspective coming into programming, you know, starting when I was 20 or so. When I was first shown VS Code and the Javascript tool chain that I was supposed to work in, and NPM and the command line, I just went, you can't possibly be serious.
Like this can't possibly be the way that I'm going to do this. And my computer looks like it's just stepped back into the eighties. Like I don't understand, you know, I've grown up in the rich graphical user interface that gives me what I need visually. And it was just shocking. So I do think that people who are programmers or professional programmers, are maybe ones that I don't know if they want to say, they're more willing to step into that world of invisible connections and implicit logic. And then it is everyone that maybe would shy away from that or would find that more stressful or just less intuitive. They end up being like, okay, well, programming is not for me.
That really resonates with my experience. And I think there's a few things going on here. One is this issue of accidental complexity, where you have aspects of the experience that are just not really essential to doing the actual programming work. Like basically the tools are designed poorly and unnecessarily so. There's too many things going on. And some of them don't work. They're unnecessarily confusing. I had a similar experience when I first wanted to do a web app at a time, the state of the art was like these Java servlet things. This is like full on enterprise Java wildness. And when I encountered that, I was like, there's no way I'm going to do this. Sorry. And so I went back to my DrScheme where everything is in a nice little bubble. Eventually you got Ruby on Rails and everything was fine and so forth, but there's this other issue of communication and information theory.
So by that, I mean, a lesson that I learned from management is whenever you're communicating something, something is said, and then something else is received on the other end, that's like a lossy process. And I think the reason, a lot of programmers like text-based interfaces is that they have very low loss there. They kind of understand what all those symbols mean. And furthermore, they're very dense, whereas a new programmer. And I was once one of these, basically it all gets lost. It just looked like a garble of like alien symbols. So one of the reasons that visual programming environments could be so appealing is they have lower underlying information density, but they're constructed in a way the optical clarity is higher, right. More information gets through to the other side. And I think ultimately that suggests that there's this end game of you get both the very high bandwidth potential of textile interfaces with the augmentation that you can get from other visual aids. I don't know exactly what that looks like. Right. But I think there's something there.
I definitely, I think discovered that when, as part of my work for Egghead, a lot of my work is illustrating programming concepts, right? So it's very much just educational material, we're talking like articles or animations that kind of go along with the video lessons where, you know, I'm visually explaining things like, you know, how GraphQL works or how React hooks work. So it's just taking concepts. And what I believe I'm doing is making visible the mental models. I tend to use the word metaphors because I'm from more of a cultural anthropology background and I've read a lot of thinkers like George Lakoff and Mark Johnson, who talk about embodied metaphors and sort of the fact that almost everything that we talk about in the abstract realm is based in our embodied experience of the world. So we have this inherent understanding that, like, up is good and is more and is happy.
We'll say the stock market went up and the stock market didn't physically move upwards, right? Like it had more money move into it, but we use the term up to think more, good, better, right. And we'll say like, you know, I'm feeling down today means I'm feeling sad. You're not physically moving downwards. But the point is that we have these metaphors that are based in being humans with bodies that have like fronts and backs and ups and downs. And we map all our abstract reasoning on top of that. And when you have to draw like programming function, you really have to look at the language we use around programming functions and really figure out like, okay, there is a physical metaphor baked in here because all things are based off physical understanding. Um, and then drawing it, you know, you draw, okay, there's a component, that's a higher order component in top of a lower order component.
It's passing data. So it's mimicking this idea of physical objects moving through space. So it was very obvious to me and trying to draw programming concepts that it's based in physical knowledge and visual programming languages or visual interfaces, just make that explicit. Right. They like actually draw a box that is above another box, or what is to the right of another box, which suggests it's in the future because we think the future is to the right. And the past is to the left, at least in English, Western speaking countries. That's how we map time. So I've always just been kind of fascinated by how we can sort of use these embodied metaphors to help people understand programming concepts, because everyone has a body. So you get these things for free. Like we all get the concept of front back and we can use those in interfaces to communicate to people, you know, the relationship of these otherwise very abstract lines of code that are just encoded things on a hard drive that would otherwise be incomprehensible.
That's super interesting. And also it actually connects to our design ethos for Muse. A lot of our concept there is we want to tap more into, for example, your spatial reasoning, which is very strong for most humans, as well as yeah, things like the direct manipulation and touch. And so on. Basically taking advantage more of our bodies' and our minds' natural, you might call them talents, or just built in capabilities. Because of course so much what we do with computers, we're really bringing ourselves into an abstract intellectual realm and we can learn to do it, but it is very much a learned and a learned over a long period of time skill.
Yeah. And I think this idea of understanding whether that's through metaphors or physical embodiment or mental models is really important because ultimately most programming errors or failures are ones of understanding and not ones of like mistyping. And I think a lot of our tools, they focus on the mistyping issues and it's much harder, yet more important to get at the ones of understanding. Like if you think about the most gnarly bugs you've dealt with, it's usually because you misunderstood something about the world you're modeling or how your program worked. Not because you typo'd something, if you did typo something that kind of gets fixed quickly. So anything that can help us understand what actually is, is very powerful, which by the way, is another reason why I'm a huge fan of logs because they are a sort of ground truth with very high information density of what actually happened in your program.
Yeah. I've actually recently been working on a project that really sort of emphasized to me how important this is, right? If the mismatch between the program, creators mental model and the person trying to write the programming language. So of course I've mentioned JavaScript because that's just kind of the world I work in. So this is a collaboration project that I worked on with a programmer named Dan Abramov, who's a really wonderful kind of well-known JavaScript developer, who's on the React core team. And so we've been collaborating to help create a course that visually diagrams and animates the underlying mental models of JavaScript the way that it actually executes. Which helps people get over a lot of the bugs that are written into JavaScript code, simply because they don't understand what happens when you say, let banana equals fruit, whatever. And then you reference that later, but if it's been reassigned or it's inside an object and it's been passed through a function, they aren't tracking in their head exactly where the variables gone and how it's changed over time, because it's not visible in the program.
So anyway, this project is all about showing people, literal, explicit diagrams, showing them what happens when certain types of code executes in JavaScript. And it corrects a lot of the mental models that are incorrectly taught in a lot of JavaScript textbooks. And this is where working with Dan and being able to help visualize this course, I completely reformed my understanding of how JavaScript code executes, but there's no IDE that was going to show me that. So it was just one where I went, it's great to make educational materials for this stuff, but it's not built into the tool.
Exactly. It reminds me of one of the earlier programming courses I took was in C and it was kind of notorious because there was this one point in the course where a lot of people hit a wall, which was when you introduced pointers, you know, it's this incredibly important and fundamental thing that totally changes how the variable behaves. But it's just the little star, you know, in front of the variable name or whatever. And it kind of resonates with what you were saying about the mismatch between how people are thinking about what's happening versus when it's actually happening and the importance of correcting that.
How do you go about coming up with a metaphor when you're working on something like the Just JavaScript course or any of these Egghead pieces, presumably you need to develop your own understanding of it either from your own experience or from the course materials, but then you want to go to translate that to some kind of visual thinking or create a metaphor, hopefully one that's correct and useful, how do you do that? Can you give us an example?
Sure. I'll say this sort of a spectrum. I mean, I like spectrums anywhere. There's a spectrum of visual programming and there's a spectrum of metaphors, I'll say. So the Just JavaScript project with Dan probably leaned a lot more on very direct, spatial metaphors. It was very much like what is connected to what, what is above what? So it's sort of circles and arrows in a very classic diagrammatic way. But a lot of the other work I've done for Egghead is illustrating programming concepts. And what I would say is on more towards the poetic spectrum of metaphor, where it's not technically like, you know, the most accurate, it doesn't exactly map to syntax showing what happens at each point, but I'll make a lot of illustrations that will be metaphors that give you an understanding of like, oh, okay. If it's a waterfall image, it's like, okay, this is something about data flow.
That sort of level of poetic metaphor that gives you more of the gist of an idea more than the little syntax interpretation. But I'm thinking of one of like, I did one for Redux, which is a state management system for React. And I use like a little joystick with space aliens, like shooting, cause it has actions. So it's like, you can like shoot out actions from Redux, from like a single state source. So that's one where it's a bit more fun. Obviously it's a lot more visually interesting than just doing a box and arrow diagram, but it gets the concept across to people that you have a central place and it's shooting actions out from it. And that the process I kind of do for that is whenever I'm given something to design a metaphor for, I will read docs, you know, I'll read programming articles, I'll go see what people are talking about on Twitter related to the thing. Stack Overflow questions. And sometimes if the community is already speaking about it in certain metaphors, I just use that because they clearly already understand it. If they're trying to explain it to each other on Stack Overflow and they're using these metaphors, that's usually what they'll base them on. But a lot of the time I'm just kind of doing basic reasoning of going, okay, what are the functions of this thing, what are its qualities. And then you do a bit more creative lateral thinking to say, okay, what else in the world has those functions and those qualities?
Yeah, I think that, that points to another reason to have this kind of illustration in learning materials, which is just to make it fun and break things up. And, yeah, visuals to break up for example, long streams of text, or if it's more of a video format on strings of speaking, or you're just looking at a screen that kind of looks the same the whole time. You know, it's just a code editor window or something. I'm reminded a bit of Why's Poignant Guide to Ruby which was a beloved introduction to programming for the Ruby language and have these adorable cartoon characters and kind of a web comic style. And you know, a lot of it was not specifically helpful for helping you learn it. It just made the thing just more fun to consume and more memorable because you connect chunky bacon with, you know, some of these more practical concepts that are being taught to you.
Chunky bacon. That's a throwback man. Wow.
Now, I can't believe we got this far without mentioning Bret Victor's seminal piece Learnable Programming. And maybe somewhat to the point you've both made in various ways, which is it doesn't have to be either or the idea of creating embodiment or visualization or showing more of happening. The abstract pieces, how things are connected, how things flow in time can also go with symbolic code and this piece from some years back now. But I think has been influential in terms of, well people lamenting, we don't have programming environments that are a bit more like this, but demonstrates lots of interesting and inspiring ways to potentially make, for example, control flow more visible.
Yeah. The Bret Victor piece has kind of been a seminal launching point for this whole visual programming industry. Of course it was alive and thriving well before. I mean you can kind of trace it back to of course, Alan Kay and Xerox PARC and Smalltalk. And there's a lot of precedence for this, but the learnable programming piece, I think is the one I've seen referenced the most and has gotten the most people, you know, maybe in the last five to ten years, more interested in it and to get them to go exploring and to go, okay, maybe the way we program now doesn't have to be the way we're always going to program. And the thing I really loved about Victor's piece, I mean, of course it's a long article and there's many aspects to it that we could kind of dive into. But this idea of just that, because code, when it comes down to it, it's something being executed in full on machine code and byte code.
And with so many layers of abstraction up from there, it's not like we only have to represent it in one way. When you get into the higher level programming languages, syntax is one way of displaying what that machine code is actually running and the rendered UI is another way of displaying what that machine code is running. And I don't know why there's not a reason why we can't flexibly display this same set of machine code in different ways and turn it on and off and you know, say, okay, I want to look at it in visual data flow setting, or I want to look at it in a way that shows me the component architecture, if it's like a componentry kind of context. So just flexibly, displaying it in different visual formats rather than saying, okay, syntax is the only way I'm ever going to interact with this code until it becomes the final output.
Yeah. Interesting. And now that you mentioned that I feel like as is often the case, the gamers are ahead of us. I mean, particularly the game designers, like I'm not an expert in this, but my understanding is that the game engine environments, like the very sophisticated, there's like the code view, then there's like the object graph view. And you can like, you know, turn on different pieces of the visual environment. And there's all kinds of performance-based views for looking at memory and CPU and GPU usage and so on. So there's perhaps something to learn from that industry.
I also think the data visualization industry is one. We could really draw a lot from like I know Bret Victor had mentioned he had researched a lot in Edward Tufte's work, who is a information designer, I suppose, is the best term for him. But he's written a number of books like Visual Explanations and Beautiful Evidence that cover kind of the best practices for how you can represent data as visual charts in a way that actually respects the original data and presents it in an accurate way that isn't bending it or being too manipulative about the way that you're it. But he has yeah, a wonderful set of sort of principles for how to do it well. And Victor had talked about how he read him before writing Learnable Programming, but the data vis community in general has done a ton of work on thinking of different ways to visualize numbers.
And they are fairly well known for it. Do you guys know of in dashboards, you can sort of turn on line chart or you can turn on bar chart. You can sort of jump through different modes of being able to look at the same numbers. And that's sort of in my head, what we might be able to do with syntax in sort of maybe IDE plugins in the future. Of course, there's plenty of IDE plugins that do help you look at your code in different ways. But more of that is kind of what I would love to see is like more plugins that just like, I'm just going to show you the architecture side of your code. I'm just going to show you how this variable changes over time in your code. I'm like one of those where you're kind of like, am I missing something? Like why isn't this happening?
Yeah, totally. I'd love to see more of it. And I think it can come both from the IDE language plugin layer, but also bottoms up from individual projects. I feel like I must've, at some point done one of these things where you write a little script that like takes your code or your system and compiles a DOT graph so they can visualize nodes and edges with one of these standard DOT visualization tools. And I feel like there's a lot of opportunity to do that in individual projects. Like you feel the need to have a different, more graphical view into your project. Well, you're a programmer, you can make it so.
And it kind of, to bring us back to what end user programming. I mean, the strange thing about programmers, right, is they are end-user programming, like of all the industries. They're the only ones where they are both the users who have the ability to actually build their own tools and to actually get in and change their own applications and no other, I guess, community that uses computational tools is able to do that.
Here you're talking about a programmer writing their own VS Code plugin, something like that.
Yeah, exactly. Or even if you use a certain language or framework, you can build a library that adds on to that, right. Even if you're not on the core team and you, aren't going to PR into the main repo of that framework or tool, you can build extensions off it. You know, it's the beauty of being able to fork and change and collaborate on these things means that there's so much flexibility and in developers having their own authorship power.
Maybe we need more, a culture of jig making and software as there is in woodworking. Woodworking is interesting because it's one of the rare fields where maybe it's the only one I can think of where they have that ability to kind of build their own tools in wood. And there there's this whole culture of making jigs to help you build the furniture. In fact, you might spend most of your time making the jigs and then the furniture is kind of this final culminating step. Perhaps we need more of that in the software world.
Some of the ways to visualize your code or your program differently that you both have mentioned like, yeah, in game world, maybe something like Unity, or you tend to build stuff into the game that lets you look at the scene graph in different ways. And I think we have that a little bit in the web world. Dev tools is one that we tend to give a lot of love to. I think we talked about that a good bit with Geoffrey recently here on the podcast, but there it is an example of what you said, you see the output, but there's also the kind of hover over elements and highlight, which is where and what the extents are. And then there's also the margins and padding stuff that kind of goes down in the lower right corner. But then you've also got a expanding hierarchal graph and you can see those all at the same time.
And even as you're mousing over, it's highlighting. So you're looking at exactly the same thing in three different ways and you can spot different problems or understand by visualizing in these different ways. And I would further argue that we see that a little bit in more the native development world with toolkits like Flutter and Swift UI, where there often is even a live, side by side, here's your interface output, here's the code. And often you can change things on both sides and it flows bidirectionally. But I think in all of those cases, it's already something where it's a visual output. And so you're interacting with some of the under the hood stuff around that visual output. It feels very natural to visualize that. When you come back to the more abstract things like program flow over time, that's where maybe it's less obvious to do that. Or maybe it's just less clear how you would do it in the first place. But yeah, I think that visual debugging or visual understanding of something that has already a visual output to begin with is actually a pretty rich place for this stuff.
And especially, I mean, you had mentioned the two way or bi-directional aspect of it is super important. I mean, one thing I'm really liking is there's kind of a new rise in state charts and state machines, but visual ones that flow both ways. I'm specifically thinking of, there's a tool called XState that works for React where you can change either the diagram or the code in a side-by-side and they mirror each other. Whereas if I want to debug some CSS thing in the browser, it's like, I'll go test out in the browser, but then I still need to go back to my IDE and like actually change the actual code. And it's just that like slight friction of you're not actually changing the thing itself. You're like mocking it up in one. And then you go back to the actual source. We'll know, and, you know, interaction design, any amount of friction you put into a tool or an interface it's just causing that much more cognitive load on the user.
I'm looking at XState right now. I hadn't seen this before. It is very cool, like code and flowchart diagram side by side. We've done a bit of research on the bi-directional editing capability as part of Ink and Switch. And it can be very, very hard to do that well. If you have something like maybe a tool like Apparatus would be a good example, I can link that in the show notes, but something where you have the ability to do direct manipulation with dragging something with the mouse or your finger in one kind of pane and the other, you have code that you can update, but having not one be the end state, the output, I don't know the bad version of this is Dreamweaver, right? The output it writes is just completely incomprehensible and you can't kind of go back from there. So that bi-directional thing I think is sort of a holy grail in one way, but also it's very hard to do technically. And I think also depends a lot on the nature of the programming language itself and the tools you use to edit kind of the symbolic representation.
Yeah I usually take the long stream of maybe I don't want to say failures, but disappointments in the WYSIWYG world, you know, everyone that's ever dreamed of, yeah Dreamweaver where you draw the website and you can change the size of the div and the code, you know, reacts appropriately the fact that no one's solved it after this long and that many probably billions of dollars. I'm not quite sure. It shows that it must be an incredibly hard problem, you know?
Yeah, indeed. And I know in the Xcode world where it ships with this interface builder, which is essentially a drag and drop GUI system for setting up your interface, very often, I've seen iOS developers. They prefer to say, just don't use interface builder, write everything in code because sooner or later you're going to be basically working in code anyways. And if you have some of your program is represented in this visual thing, that's edited one way, some of your program is represented in code the two interact with each other in a bunch of complex ways. You'd rather have just the one medium. It's actually better longterm. Now, another great article that I came across by total coincidence just as you and I, Maggie were talking about possibly doing a podcast on this topic is an article that is called Hacker News folk wisdom on visual programming and it essentially is a pretty large scale analysis of every single time something visual programming related has come up on Hacker News and sort of like distilling out the common themes and then starts with the really common and fairly shallow reactions. And it goes into some much more nuanced and interesting discussion. So there's lots here, including some we've already talked about, but is there any pieces of this article that are especially interesting to you? Maggie?
It's funny. I had been following a lot of those same threads and there's a fair number of them and they're the sort of Hacker News topics that go for hundreds of comments, right? People just go deep on this stuff. I had been reading a lot of those same ones and passing through them, right when that article was published and I tweeted the author, I said, you know, thank you so much for doing the hard work of distilling that all for us. But I think the thing I found most interesting about it and just from reading the threads is, I mean the strong opinions people have on it, or you really get into people have very emotional, I want to say like, reactions to visual programming and sort of the cultural anthropologist side of me just finds that quite an intriguing thing to look at where many of them, I want to say have almost been like emotionally scarred by past visual programming languages, like the stories they're telling in the comments of the time they had to work on the most horrendous proprietary visual programming platform.
And no one could find anything. And the literal spaghetti code is just all over the place and they just wanted to go back to text. So I think there's some very legitimate critiques that should be acknowledged that maybe the past generation of programmers, was sort of subjected to a really awful version of visual programming. But it would be really sad to let that frustration that other people have felt ruin what visual programming could be in a much better design format or just done in very different ways done in ways that combine graphics and text. Yeah, so that was my main takeaway from it is almost like you have to do some historical healing before you can kind of move forward with this concept.
It is interesting that it can be such an emotional topic because I think if we were to boil it down to a bit of a duality, you have both people, like, I think those of us here talking right now who really believe in some kind of end-user programming future or programming being more accessible or just computing as a medium is so important for humanity's future. And we need just more humans and a broader array of types of humans, different perspectives and different backgrounds to be able to access this medium. And so that's maybe something we feel strongly about. Certainly I feel very passionately about. On the flip side is maybe as you said, first of all, the bad experiences that have scarred people, but also you do come back to this again, the Hacker News comments are people who many and most are probably professional software engineers.
They are people for whom the pure text model and classic text editors and IDEs and so on serve well, or maybe it's less about being served well. And more that they've had to go through the pain and suffering to get over that hump, to learn all those abstractions and managed to hold all that stuff in their head. And then in a way maybe they would feel robbed if someone else can just kind of skip over that. I actually had that feeling. I learned a little bit of electrical engineering many, many years ago, doing Burning Man art projects. I wanted to make stuff that lit up in cool patterns with LEDs and so on. And I had to go through the whole kind of classic electrical engineering process. And I found it very challenging. It didn't really fit with how my brain worked. I thought it would be relatively easy because I'm a computer programmer, but circuits are a whole other thing.
And just a few years later, we had the Raspberry Pi and the Arduino and increasing number of these kits and Adafruit and all these much more accessible things. I remembered talking to a friend who said, oh, you know, I'm going to set up my project that will have, you know, these LEDs or whatever, it'll color cycle like this. And like everything you're describing there, that's going to be a lot of work. I hope you really budgeted some time. And I don't know he bought these kits, you know, off Adafruit. And a week later he had this amazing thing working. And I went in and asked him about it and realized that he just didn't need to learn all this stuff about debouncing with capacitors or whatever that I had to fight through because there are these just more accessible platforms and there's a knee jerk, I think I had that then, a knee jerk kind of like, wait, I had to walk uphill both ways in the snow to learn to do this other people should have to too, or I wasted my time or something like that. But of course, I think it's much better that there are much more people that can now do things like make cool light up projects because these technologies are more accessible. I feel like there's probably a version of that for today's programmers versus people that might access it in the future. If it was a little bit more learnable.
Definitely. There's also another, I suppose, trend in people that are a bit anti visual programming, or I'll say resistant to it, or have some, you know, skepticism and reservation around it. Is this, I have found this sort of a duality where people lump themselves into going, okay, I'm a math logic person and art people are on the other side of campus. And I am not an art person and everything visual is art. And this idea that they're not visual thinkers. I hear that all the time from people who are in programming and I can't really fathom what that means in a lot of cases, because I go, well, visual is about embodiment. You have a body. And I think, you know, you could probably draw a box and an arrow and communicate some visual stuff. If you weren't trying to draw some detailed drawing. But yeah, I think the, the, I'll say historical divide of art from science and visual from logic, and this idea that math happens in textual syntax is a bit of a hangover that people just think, well, visuals don't belong in math and programming, but I mean, this is why I think cultural anthropology, one of the sort of points of it is to sort of show people the vast diversity of expressions that human life and culture can take.
And that everything we have was invented at one point in history. And we are one very specific narrow manifestation of possible human culture. And there are many others and we could switch to them or move towards them. And we don't have to live in some world where math and art live on alternate sides of a dichotomy.
That's a really powerful point, Maggie. I feel like we should wrap it up there, but for people who are wondering about examples, they want to see what this looks like, because indeed it is a visual topic beyond the ones we've already mentioned, are there some good resources you can name that allow someone to flip through and see the variety of different experiments folks have tried for visual programming?
Definitely. There's some people who have done a lot of wonderful labor compiling screenshots of, I think every possible visual programming language or platform that's ever existed. And one I want to point to that, we'll put in the show notes is a Visual Programming Codex that Ivan Reese put together, which is a GitHub repo with just a wonderful array of examples. And there's another one called Whole Code Catalog. So a nice play on the Whole Earth Catalog that has again a nice sort of categorized collection of different programming platforms and alternatives and the sort of the different qualities each of them has all nicely organized in a beautiful website.
Well, thanks everyone for listening. If you have feedback, you can write us on Twitter at MuseAppHQ, or on email, hello@museapp.com. You can help us out by leaving a review on Apple Podcasts and Maggie. I'm really glad you're out advocating for making programming more visual, more embodied, more learnable, and I'm especially inspired by the idea of having a less bifurcated world between the art thinkers and the math thinkers. I think we need that. That's a beautiful vision for me.
Thank you so much for having me on. I really appreciated getting to come on the show and I've mentioned to you, I adore the podcast. I think I've listen to every episode up until now. So it was really wonderful to get to finally come be a guest.