Few seats left for our Advanced Backstage Plugin Development workshop on December 7th. Secure your seat today!
CHARLES: Hello everybody and welcome to the Frontside Podcast Episode 47. I'm Charles Lowell and with me today is Noel Rappin.
Before we get into who you are, because I know that a lot of people know you already and you need no introduction, you're actually calling in from Chicago today, right?
NOEL: That’s true, yes I am.
CHARLES: Did anyone actually show up for work today?
NOEL: Yeah, people did actually show up for work a little bit later. So, this is being recorded the day after the Cubs won the World Series last night. I will say that my commute in the parking lot at the commuter rail station in the suburbs was surprisingly empty this morning. The people were a little bit slow to come in. The game didn’t end till about midnight last night.
CHARLES: I'm neither a Cleveland nor a Chicago Cubs fan but it was a harrowing game. It almost gave me a heart attack.
NOEL: Oh, man.
CHARLES: I can't even imagine what it was like.
NOEL: A Cubbie Blue since I was a kid. So yeah, it was something.
CHARLES: So, you're from Chicago?
NOEL: Yeah, I grew up in Chicago. I grew up in the Chicago suburbs. I went away for school and I worked in Boston for a while, but came back about 12 years ago.
CHARLES: Okay. So, that kind of leads me into the perfect setup for one of our questions is how did you get into development? I've known about you since I got into the Ruby community, probably back in 2010 or something like that. How did you come to be there?
NOEL: I was one of those kids who was a developer at a certain age, I guess and I had an Apple II. I started programming Applesoft BASIC in the mid-80's, I guess, and eventually decided that I enjoyed it enough and got pretty over-educated at that and that’s just a very traditional background.
I went and worked at what we would now call web consulting company around 2000, my first professional job was '98 - 2000. I had done a lot of different languages as a student. That job was mostly in Cold Fusion and then eventually bounced around doing a bunch of different web development solutions.
At some point along the way, I picked up the Ruby Pickaxe book and then eventually came to Rails. I started Rails as a hobbyist, either pre-1.0 or very early 1.0. I used it to build some task tracker for my team and a couple of other little tools like that. And relatively shortly thereafter, I started doing it professionally.
CHARLES: I think you're kind of the same generation that I am. I call this [inaudible] from that early 2000's web development background where it's kind of fueled by [inaudible] network.
NOEL: Yeah, a little bit. My first professional job was at a company that is a very, very small consulting company based in Miami and Boston and they were not developers who run the company. They’ve been documentary film makers and there's like a whole 90's technology trajectory here because they were documentary film makers and then they were documentary CD makers and then they were asked to add interactive stuff to the CDs that they were putting out and then they became web developers as a sort of a natural extension. And suddenly, it was like 1998, every company in the universe wanted to be on the web and they were like, "Oh, this seems like a good business to be in." And they started trying to hire actual programmers who knew what they were doing even though I was just out of grad school and didn’t know what I was doing but it was convincingly put in.
We did all the stuff then that would be like the first application that I worked there, the production server lived under my desk and was also the staging server and the development server. We didn’t use, at that point, search control although eventually we did. There were no established conventions on what was the good thing to do.
CHARLES: Yeah. The production server under the desk: a more elegant weapon for a more civilized age.
NOEL: [Chuckles] Certainly old.
CHARLES: That was a while back but you’ve obviously been doing a lot more structured, a lot more formal development over the course of your career. So, most of us, myself included, were kind of content to write the software, maybe blog about it every once in a while, give the occasional conference talk, but you’ve really, really delved in to those activities and really kind of taken the time to really, for the topics that you're interested in, really kind of research them very thoroughly to the point where you’ve actually written several books. When I think about the concept of writing a book, it sounds like, "Oh, my goodness! That sounds terrible. That sounds like a hundred of my teeth pulled out," or something like that. But you’ve done it multiple times. I'm curious, did you experience that kind of fear and uncertainty and what was kind of the decision making process that led up to, "No, this is something that I've got to do. I'm going to sign up for this. I'm going to do this. And I'm going to make this commitment."
NOEL: Fear and uncertainty is a different question. I've always written things. I was a writer in some ways even before I was a programmer. So, being able to combine the two is something that I really enjoy doing. And I've always really enjoyed teaching. The more cynical way to doing that is I enjoy explaining things in a way that makes me sound smart.
But I've always enjoyed doing it in whatever context. So, when I'm not writing, I've always often been involved in training at the places that I work or other things like that. And I've always been a little bit of a theater background and a little bit of a performance background and conference talks are a way to still scratch that itch a bit.
CHARLES: You mentioned that you have a graduate degree.
NOEL: No, my graduate degree isn’t Computer Science but I did a lot of amateur theater things in high school and college. It's not something that I have a whole lot of other outlet for other than talking about development.
CHARLES: [Chuckles] You wouldn’t think that it's a natural thing but I think the best talks and the best walks. What's it like to have that element of theatricality to them?
NOEL: I try in the books to give them a little bit of a voice. Sometimes you get interesting responses when you try to make a technical book a little bit funny. Some people really don’t like it and other people really respond to it. I do it sometimes just as a way to maintain my own interest in the 14th page of this chapter or something.
As far as fear and uncertainty, that never really goes away. I have a book out that I'm writing now about payments which is a very serious topic. And as much as I've done work in that and as much as I've done research on that, there are obviously people out there who have done more work and know more certainly about individual pieces of it. And there are a lot of people who are going to take anything that I write and implement it that way. They're going to take it as the way that things are done. So, that does introduce a tremendous amount of unsettling.
CHARLES: Yeah, you're on the hook. They're like, "But Noel told me that this is the way to do it and I ran out of business."
NOEL: Yeah. Actually, the payment book which is called 'Take My Money' is actually only a second book in the history of Pragramatic Programmers to have a legal disclaimer at the beginning which I'm very proud of.
I think that the thing that is the most worrying especially when writing something long is not so much factual errors because I can check facts. Facts can be checked. Emphasis is much, much harder to check and the cultural community practices are much, much harder to check. And so, I'm often much more worried about the idea that I might spend a lot of time on something that nobody really is interested in or that I might not spend time on a really important thing just because it's a part of the world that I don’t know as well as other people in there, and therefore, I miss something. Factual errors, I worry about them too. But factual errors are relatively easy to catch often, not always.
Many, many years ago, I wrote for Rock's Beginner Rails book which was purchased by approximately nobody. But it's got one of those beautiful rocks black and white cover pictures of the author on it looking terrible. At that point, I had worked in Rails professionally but only for a very, very short time and there were certainly people who had a lot more experience than I did. But for whatever reasons, I had the book contract. In that context, there's a lot of uncertainty around not being sure that I know what the newest tools are or what the best practices are.
CHARLES: How do you mitigate that? How do you address that?
NOEL: Research is one way. There are pluses and minuses to not being the greatest expert or something when you're writing a book like this. One of the pluses is that your experience is a little bit more immediate and it's a little bit easier for you to make it relevant to somebody coming to it for the first time. But then a lot of times, you wind up doing the kinds of things that you might do if you were actually trying to learn it. But then I am a pretty good researcher and then I can compress that down into something that took me 4 or 5 hours to Google or research, or days to figure out the best practice, or talk to some experts sometimes and then I can hopefully distill that down to something so that you don’t have to go on that entire journey. You can just cut the chase a little bit.
It's researching. It's finding the people who do seem to be prominent in a particular technology or a particular skill and finding out what they're doing. Sometimes, it's getting it wrong, hopefully not very often. And sometimes, it's just like this is the best I can do and I have to like, "I can't solve every problem."
CHARLES: Right, just being satisfied with that.
CHARLES: I get it. That makes a lot of sense. You mentioned that the book you just wrote, the one about taking payments on the web, I saw that. I looked at it and I was like, "Oh, that totally needs to be this book. This book totally needs to exist." And thanks that it actually got written.
But in hindsight, it's obvious that there wasn’t really anything that like it treated web payments as this holistic topic that needed to be addressed and yet, when I look at the last 5 years or the last 10 years, this is a body of knowledge. It's been part of 90% of the applications that I've developed and it's been something inter-rolling like they’ve been patterns and stuff developed around it but I've never really thought about it, "We need to draw this circle around this practice." And this needs to be researched and there needs to be at least some guide on how to do this.
And so, I think from my perspective looking in hindsight, seeing why it was published, "Oh yeah, that’s totally obvious that that should have existed." But being on the other end, looking into the future of, given the infinite number of things that I could write about or I could research, I need to go about kind of the selection process and say, "This is something that really ought to exist."
NOEL: This came about kind of interesting to me, I guess. The purpose of the book states it out pretty clearly. I, a few years ago, started working on - not a large web project by any means, but certainly one that had a fair amount of complicated business logic around their payments. And we inherited it with the payment gateway in place and I kind of thought that, "Oh, the payment gateway is there. That’s the hard part." We just basically sat and pretty quickly learned that that was not the case.
As I was really looking for best practices, things people who had delved to more problems on the assumption that a lot of people had to solve some more problems. One thing in particular that I was looking at was handling the failure case where I have to do something in my database and I have to process the credit card and they can't happen simultaneously but the failure of one affects how I process the other. I'm just looking around for how other people had solved that and really being surprised how little information I found about something like that.
I kind of filed that away. About a little over a year ago, I had this weird idea that I was going to start doing some self-publishing again and sooner writing about this or thought about writing this as part of that, writing about payments and data modeling and something like that. And as I started getting into it, I was like, "I had enough issues here. This is longer than like a blog post and this longer than a magazine article. There's kind of a lot here."
I worked with Pragmatic, so I know the editors there and I sent them a proposal and they pushed back a little bit on the idea of, "Aside from the documentation, what else do you need?" And I was like there are all these issues here. They know, Dave, they run their own web store, they wrote their own code and I actually was - part of when I got the contract because I got to interview Dave Thomas about how the Pragmatic store works which was great. It was really neat.
CHARLES: Did that material actually make it into the book itself?
NOEL: Not as such because I was a little reluctant to directly quote him about the internals of the Pragmatic store, but some of the principles Dave talked about. It was actually kind of reassuring that about 85% of it agreed with stuff I already thought I knew, and about 15% of it sort of went beyond that. They have some problems that are -- because they're actually managing physical inventory which a lot, not all web stores do, which introduces a whole host of other problems.
It became a combination of I actually really do think I've worked on a lot of these problems and they're interesting. And a lot of people seem to have similar problems and there's now a whole lot here, and so it seemed like a book was a way for me to go on that. Other people might have gone a different direction.
I am getting kind of an interesting reaction from people. I get a lot people who are saying, "Well, I really could have used that book two years ago." And not a lot of people say, "I really need to buy this book right now," which indicates maybe the marketing needs to be tweaked a little bit. I'm pretty sure it will. It's not even fully out yet, so it will find its audience. But it’s interesting how I think people still even seeing the book is there assume that they know more about what they're getting into when they start doing payments. And then they come out the other end a couple of years later and they're like, "Oh wow! I really wished I had the guide on that."
CHARLES: So, what you need to do when marketing this, you need something along the lines of 'you don’t need this book two years from now' or like 'don’t be the person who wishes they read this two years from now, read it now'.
NOEL: If I had this book when we started, I would have saved so much time and not made so many mistakes.
CHARLES: Yeah. It's always interesting to me the interplay between kind of the patterns evolving and documenting the patterns and how eventually that kind of precipitates code that makes those patterns automatic. But it feels like there's always this kind of knitting edge where you have to kind of explore the problems phase and do the research and then eventually you can kind of turn and digest and write little pieces of it, but eventually it becomes like code.
Have there been actual code like plug-ins or stuff that has fallen out of this, or that people can reference directly?
NOEL: Yeah. Not as such. There's a reference application in the book, but I submitted a couple of patch requests to a couple of gems along the way but nothing significant. It's pretty unusual for me to co-model a book like this with a lot of new open source tools. I'm usually more about using the things that are already there. But the book definitely has an opinionated set of thoughts about how you design a Rails application and it's pretty upfront about that. We're going to put all our workflow logic in their own objects and we're going to not put a lot of stuff in the active record models and we're going to do it for these reasons.
The first chapter of the book actually is not even payments. It's just shopping carts and a way to talk about the data model and the design because ultimately, money has some word aspect but a lot of it is just principles of doing good software design, encapsulated software design, and the way you will deal with any big external 3rd party dependency. But then on top of that, you have the specific pieces of money is weird, like money has real world implications and has legal implications and things like that.
CHARLES: Yeah, definitely. Whenever you're dealing with money, there's an elevated level of pressure that seems like it's put on. And I know personally when I think of -- lately, I was writing my very first payment system. We actually maintained for a while the stripe-rails gem which had moderate popularity but we don’t maintain it anymore. But it was that project where we did that where I kind of became an information recorder. And it kind of started me on this journey of never destroying information, kind of that immutability and like functional programming and stuff like that because I was so worried about over-writing any information about the payments coming in. I was like, let's capture it from the web hook and write it to the database and replicate it and back it up and let's never throw away a fact.
NOEL: The specific advice in the book is save as little of your user's personal information as you can and as much of the transaction information as you possibly can. It definitely recommends saving the entire response in the payment gateway. It recommends making sure that you store all of the pieces of your partial price calculation so that you can recreate it even if the logic changes because that is something that has specifically bitten me in actual world.
CHARLES: Can you elaborate on that when you say partial price information?
NOEL: Yeah. The application that I work on, it sells tickets. And it has a processing fee. Originally, the processing fee was whatever the processing fee was and I hard-coded it as a constant in the code base because I thought I do not need to write this processing fee to the database for every individual transaction because I was wrong. And eventually of course, they changed the processing fee which would be fine if you never had to look backwards at all and try to run a report of old transactions and try to validate that the prices on old transactions still held.
So, they changed the transaction fee and then they started running reports and all of the reports are off by whatever the changed amount was.
CHARLES: Right, I see. So you could still maybe store that constant in the code base but you have to at least write out what it was at that point when you did the transaction.
NOEL: Yes. Not all of this has managed to make its way back to the legacy code mass that I'm actually dealing with. But what I would recommend is saving all of the individual pieces. Unit price, any processing fees, any shipping fees, any tax fees, any discounts, saving all those as they were calculated as part of the transaction so that you always have them and they're not dependent on the code or the logic staying the same because all of that logic will change.
CHARLES: I'm all about the information hoarding. I think it's useful stuff. [Chuckles]
NOEL: I even started recommending originally a failed transaction was in a database transaction so it would go away. But I think that it's a bad idea. I think that you need to save -- whether you actually save a partially failed transaction in your database or whether you log something, the information of those failed transactions is really important and you want to make sure that you hold on to that stuff and you can go back and look at it. The idea that a bunch of transactions are always failing and you don’t know about it, that’s problematic.
CHARLES: It sounds like you're still working on this application which [inaudible] the book.
NOEL: Although I'm a lot more critical of the code now that I've gone through -- the book made me sort of think about this stuff a little more systematically that I had and made me make a lot of things concrete that weren’t quite in the original application. So, the code in the book is actually better than the code in the application.
CHARLES: Of course.
NOEL: It's also a lot simpler.
CHARLES: Exactly. [Laughs] Because it doesn’t actually have to do the real work, it just has to capture the principles.
NOEL: It just has to be a good enough façade to show the ideas.
CHARLES: Right. So, you're still working in this application that kind of service [inaudible] for the book. But I'm curious then how you manage that because for me, writing a blog post is hugely time consuming. Writing a book must be even more so. Do you just have it dialed in that well that you can just bang it out or do you mix doing the book writing with working on the application because obviously, it sounds like both are kind of crucial to each other. You’ve already said, "The fact that I wrote the book is actually informing the way that I structured the code that made me write the book in the first place." And so, how do you interleave those activities to make a continuous workflow?
NOEL: There's a separation there because the book is the thing I do not at work, it's my side project effectively. I'm not ever mixing them in the sense that I'm really going back and forth between one thing and the other. The application is the thing I do at work; the book is the thing I do at home. They inform each other, obviously, but separating them is not usually a problem. They're not the thing I do at the same time, so they don’t relate to each other that directly.
CHARLES: I see.
NOEL: It's time consuming but it's what I do instead of maintaining an open source project or doing some other community thing in what would be side project kind of work.
CHARLES: So you just do it on your spare time.
NOEL: Usually, the book is written between 10 and midnight for people who are reading between 10 and midnight.
CHARLES: [Laughs] I like that conception. Do you think it would work at all if say, with your job there were some amount of time allocated for writing the book because I think there's some interplay and hopefully, it's [inaudible] interplay between the two activities.
NOEL: The way my work job is structured right now, I actually do a certain amount of time for blogging but that’s company blogging. It wouldn’t necessarily cover working on the book. It would certainly go faster if I had time because certainly trying to squeeze some parts of this into my allegedly spare time becomes challenging. Writing a book like this is not just writing a book. You're actually writing to some extent an application and it's not a fully featured application but it does have to work and the tests have to pass and things like that. It has to work well enough so that if somebody were to copy it, it would run. That can be frustrating when to come to writing the book and find out that, "Some gem is updated and now, there's some weird test failure in my book," is sometimes a little frustrating.
CHARLES: Yeah, I know. I can imagine. You said that the book, most of the examples are in Ruby on Rails. Would you see it being valuable to someone who was not doing Rails development at all, maybe he was doing something in Python or Clojure?
CHARLES: Were you explicitly thinking of how the developers with the web payment problem in general is your audience or were you really thinking this is more for Rails people or where did the kind of needle fall there?
NOEL: The tricky part here is that it becomes very, very hard to write a book simultaneously using the Rails API and the Python API. It's very confusing. It's a 300-page book as it is and you have to make some decisions there. While I would like it to be valuable to anybody who's doing web payments as sort of a practical matter, Rails and the Ruby API are the ones I know best and that’s where the examples are because they have to be in something and they can't be in everything.
CHARLES: I remember a lot of those books that I read that used Java kind of as the one [inaudible] of the system that they were talking about. But it really was like, you're right, there's a line that we have to tread there because you can't just wave your hands and talk in one piece abstract principles that have no ground in reality. But at the same time, by giving concrete examples, you may limit the potential audience. I do think there is a path in there where your examples are accessible to people from different backgrounds.
NOEL: Examples are really hard, actually. It's a very tricky thing to have an example that is complicated enough to get the point across but not so complicated to get buried in extraneous details.
It's really a problem actually when you try to write about testing because almost any problem that is simple enough to be an example in a book is not complicated enough to show why test-driven development is a good idea. [Inaudible] has the same issue a lot of times like you start bringing these relatively robust methodologies to this little toy problems and it's hard to get across why they're valuable when you're working on a toy problem. Not impossible. There are people who do it.
CHARLES: If I can count the number of tutorials that were like modeling cats and dogs and people, it's like I've never written a single class that even remotely resembled any of that stuff.
NOEL: You're bringing in more complicated problems but then you become so sensitively dependent on the weird crooks of whatever problem you’ve chosen which can be difficult if you make it feel like it's less universal. If the idea is that these techniques or these tools are going to help you manage the complexity in your application, then you need to show an example that actually has enough complexity for them to sort of work. It can be really challenging.
CHARLES: Like I said, I think it's a far larger attempt than anyone cares to acknowledge.
And then we get to the point where you need a framework that actually has data binding even if it's a small framework like Vue.js or something like that, then you have some habits that will help you as you start building them.
NOEL: Your transition to using one of those tools is going to be a lot easier if you already in habit of separating out your display logic from your actual logic.
NOEL: Yeah, I still pick up Rails for new applications. I think that, especially because the ecosystem is still so far ahead of all the newer tools ecosystems, I still think it's still the best way for a small team to act like a big team. We're definitely investigating other stuff, other tools.
All of our Table XI developers are trying to pick a framework that they don’t use everyday ideally in a language that they use every day and build a common app in it just to try out some new muscles and get a sense of what the strengths and weaknesses of some of these other tools are. We're starting to bring them in slowly to our server and client side repertoires where they're appropriate.
I think that Rails is great but it's not perfect which leads the possibility that there will be something better eventually. I think it will take a while for something like Phoenix and Elixir to build up the ecosystem and the number of 3rd party tools that the Rails community has.
CHARLES: We've been playing around with a bunch of different tools here. I actually started a side project and I was using Rails and I was blown away. I was like, "Oh, I forgot how easy it actually was." And that actually can't be discounted and things like speed and things like this lightweight feeling, you can [inaudible] for the lack of an ecosystem. I certainly made that mistake with Node.js. NPM installs at the very beginning were super fast and that was the reason that you weren’t installing anything at all.
NOEL: Rails still prioritizes a certain kind of developer experience in a way that is really great for projects up to a certain size. I think there's a significant dispute of where that certain size is.
I find the Node ecosystem honestly really complicated and confusing relative to the Ruby ecosystem. And some of that is just my lack of experience with it.
CHARLES: We're [inaudible] everyday and it's complicated. It's just that it's huge. It's just mammoth compared to the Ruby ecosystem and I don’t think it started out that way, but it certainly is like that today.
CHARLES: Well, that’s about it for our time. Thank you so much for coming and talking with us.
NOEL: Thanks for having me.
CHARLES: Yeah. The name of the book that you're currently writing is?
NOEL: It's 'Take My Money: Accepting Payments on the Web'. It is available at PragProg.com. And then you can find anything I'm doing at NoelRappin.com. Keep an eye out there because there's going to be a new project that hopefully will start by the end of the year. I'm a little less confident that it's going to than I was, but I'm still hoping. Also you can find me on Twitter @noelrap.
CHARLES: All right, fantastic. Bye everybody!