Luke Melia, Aaron Chambers, and Mattia Gheda john Taras and Charles to discuss all things deployment!
Luke Melia: Luke has been working with Ember since it was under early development as Sproutcore 2.0. Ember.js powers a SaaS company he co-founded, Yapp, and they funded their business for a couple of years doing Ember consulting under the Yapp Labs moniker. They’re full-time on product now, and his engineering team at Yapp (currently 3 people) maintains around 6 Ember apps. Luke helps to maintain a bunch of popular addons, including ember-cli-deploy, ember-modal-dialog, ember-wormhole, ember-tether, and more. He started the Ember NYC meetup in 2012 and continues to co-organize it today.
Aaron Chambers: Aaron Chambers: Aaron is the co-author of EmberCLI Deploy and is currently an Engineer at Phorest Salon Software, helping them move their desktop product to the web platform. He's been using Ember for 5 years and maintains a number of plugins in the EmberCLI Deploy ecosystem. Aaron loves trying to work out how we can ship JS apps faster, more reliably and with more confidence.
Mattia Gheda: Mattia is a Software Engineer, Ember hacker, Ruby lover and Elixir aficionado. Currently he works as Director of Development for Precision Nutrition where Ember, Ruby and Elixir power several applications. He loves meetups, organizes Ember.js Toronto and co-organizes Elixir Toronto.
Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at firstname.lastname@example.org. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.
This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.
CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at the Frontside. With me also co-hosting today is Taras Mankovsky. Hey, Taras.
TARAS: Hello, everyone.
CHARLES: Today, we have three special guests that we're going to be talking to. We have Aaron Chambers, Luke Melia, and Mattia Gheda who originally met collaborating on fantastic open source library that we, at the Frontside, have used many, many times that saved us countless hours, saved our clients hundreds of thousands of dollars, if not more. ember-cli-deploy. We're gonna be talking not about that library in particular but around the operations that happen around UI. So, welcome you all.
LUKE: Thanks, it's great to be here.
CHARLES: Like I said, I actually am really excited to have you all on because when we talk about the platform that you develop your UI on, something that often gets short shrift in communities outside of the Ember community is how do I actually deliver that application into users' hands. Because obviously, we don't want it to be working just on our laptop. We want it to be delivered to our users and there are myriad ways that that can happen and it's only gotten more complex since the last time we talked which must have been like three or four years ago. I kind of just have to ask, I think that what you all were talking about then was cutting edge is still cutting edge now but there must have been some pretty incredible developments like in the last three or four years. What have been kind of the new insights that you all have?
And so, what we decided to do was to try to evolve ember-cli-deploy into a platform essentially, an ecosystem that could let people mix and match plug-ins to do in their organization without locking them into an opinion that might simply be a non-starter in their org.
MATTIA: Yeah. I pulled some numbers just yesterday and we have currently 150 different plugins attached to themselves, to different parts of the pipeline. So some of them are about how to build the assets, some of them are about how to compress them, some of them are about shipping. And they allow people to ship it with different ways like we are seeing [inaudible] with just simply Amazon APIs or Azure APIs and some of them even are about just how to visualize data about your deployments or how to give feedback to the user about what was deployed and represent the information.
And then this is kind of a bit more detail, probably specific to us but we also added this idea of plugin packs. So, in order to help people define their deployment story, we created this ideal of plugin packs. Plugin packs are simply group of plugins. So, plugins grouped together. As a user, if you want to implement what we call a deploy strategy, you can simply install a plugin pack and that will give you all the plugins that allow you to deploy in a specific way. And that's kind of like an optimization that we added just to make it easier for people to share deployment strategies, share ways of deploying applications.
CHARLES: Right. It's almost like an application within the framework.
MATTIA: Yeah, exactly. But to stay on the community side, I think that the interesting part about what Luke was saying which was a great success for us is that all we maintain as the core team for this project is the core infrastructure, so the pipeline and a thousand of plugins. Everything else is community-based. And often, even in my day-to-day work, I end up using plugins that I didn't write and that I don't even maintain. But because the underpinning of it were designed especially by Luke and I are flexible enough. It just like has been very, very stable and very, very reliable for many years. So, I will say definitely the idea of like in the spirit of what Ember is, I guess, creating a shared ecosystem where people can add what they want and extend what was provided has been the one single biggest win of this project.
CHARLES: One of the things that I'm curious about is we've talked about how you're allowing for and kind of embracing the fragmentation that happens in people's kind of the topology of their infrastructure. What do you see is the common threads that really bind every single good deployment strategy together?
CHARLES: It's good to think about those things as soon as you possibly can and make sure that you have all three of those bases covered really before you start adding a whole ton of features.
TARAS: Sprint 0, right?
LUKE: In Agile, we call Sprint 0 the phase the thing you do right in the beginning. You've got a skeleton of an app and then you get the deployment infrastructure going, you have the test infrastructure going, so that there is no task within your actual feature development where you have to do those things. And I think that can be a valuable concept to embrace.
I would just add to Mattia's three points that for deployments, to me, some very simple qualities of a good deployments are repeatability. You need to be able to reliably and consistently run your deployment process. Sounds simple but there's plenty of operations that have run up way too long on manual deployments. So, we don't want to see those rollback capabilities if you have a deployment that you realize was a mistake right after it gets into production. I'm sure none of us have ever experienced that.
CHARLES: That never happens to me.
LUKE: You want to have a method to roll that code back. That's something that can be remarkably complex to do. And so, having some guardrails and some support mechanisms to do that like ember-cli-deploy provides can be really useful. But whatever your approach is, I think that's a necessary quality. And then I think we start to step into kind of more advanced capabilities that a good deployment architecture can provide when we start to think about things like personalization, A/B testing, feature flags, these kinds of things. And that requires more sophistication, but you cannot build that on a deployment foundation that's not solid.
AARON: I think for me, one of the things I've been really thinking about a lot lately, it's a bit of a mindset shift, I think, to get to where the things Mattia was talking about separating those different parts of deployment. And so, I really start to realize the traditional mindset around deployments like I build some stuff and I ship it to the server and then the users get it. But if we can actually stop and actually split our understanding of deployment into two separate phases. One is the building and the physical shipping of the files; and the other one's actually making them available to people. You open up this whole world of our features that you wouldn't normally have. So to be able to actually physically put stuff in production but not yet have it active, as in users don't see it yet but you can preview those versions in production against production databases. And then at some point after the fact, decide, "Okay, I'm now going to route all my users to this new thing," And to be able to do that really easily is massively, massively powerful. And so, to me, the thing I've been thinking about lately is it is a small mindset shift away from packaging everything up and pushing and overwriting what's currently there to being something, again like Luke said, immutable deployments where everything we build and ship sits next to all the other versions and we just decide which one we want to use to look at it any time which leads into then, I guess, A/B testing, feature flags and things. So I guess deployment really is not so much about the physical shipping, that's one part of it. To me, deployment now is shipping of stuff, as in physical deployment and then the releasing it or enabling it or activating it to users.
CHARLES: Or routing it. Sounds like what you're describing is an extraordinarily lightweight process.
AARON: It is, yeah.
CHARLES: To actually route traffic to those files.
CHARLES: I mean that's absolutely fascinating, though the capability that you have when you have the ability to have these versions, the same versions or different versions of your application sitting along next to each other and being able to route traffic. But it also seems to me like it introduces a little bit of complexity around version matching because only certain versions are going to be compatible with certain versions of your API. You have different versions of your API talking to the -- so the simplicity of having kind of mutable deployments, so to speak, is that everything is in sync and you don't have to worry about those version mismatches. Is that a problem or this could just be me worrying about nothing? But that's kind of the thing that just immediately jumps out to me is like are there any strategies to manage that complexity?
CHARLES: Keep repeating ourselves if we think that that [inaudible] actually doesn't exist even in the simple case?
MATTIA: Yeah. I think that that's to reiterate what Luke is saying. That's exactly the point. You can pretend it's not there but it is and you have no way to avoid it. Once you ship something to a browser, you have no control over it anymore. And so, you have to assume that somebody is going to be using it.
LUKE: Aaron, I think you too, I don't know if you can share it. But you recently told us some stories about kind of what you encountered in your work about this and of how long people were using versions and stuff.
AARON: Yeah. Something that we hadn't sort of put a lot of thought into. But the last place I worked at, we had quite a long lived app and we're using feature flags and we're using launch [inaudible] something and it gives you a list of flags and when they were last requested. And there were also flags that we removed from the code and it was just a matter of waiting until all the users had the most current version of the app and weren't requesting the flag anymore. But this one flag just kept getting requested for months and we just could not work out why. It really sort of opened my eyes up to this exact problem that these long lived apps set in the browser and if you have someone that just doesn't reload the browser or restart the machine or anything, your app can live a lot longer than maybe you actually realize it is. So we're shipping bug fixes, we're shipping new features, and we're all patting ourselves in the back. We fixed this bug but have we really? If your users haven't reloaded the app and gotten the latest version, then you haven't actually fixed the bug for some number of people. And it's really hard to tell them as you think about this and put things in place, really hard to tell what versions are out in the world, how many people are using this buggy version still.
CHARLES: Yeah, that's an excellent point. I haven't even thought about that. I mean, what is the countermeasure?
AARON: We hadn't made it until we came across [crosstalk].
CHARLES: It's nothing quite like getting smacked in the face of the problem to make you aware of it.
AARON: That's right.
CHARLES: So, what's the strategy to deal with that?
AARON: I guess for me, my learnings from that would be from very early on thinking about how we're going to encourage people to reload, to start with, and maybe even have the ability to force a reload and what that means but then that has gotchas as well. You don't want to just reload something when a user is in the middle of writing a big essay or something like that. But definitely thinking about it from the start is one of the things you've got to think about from the start. But I guess something that I'd like to implement and I've kind of thought through but not really explored yet, but the ability to see what versions are out there in the world and there are things I've been thinking about in terms of this little server that serves different versions. Maybe we can start having that kind of tracking what versions are out there and who's using what and being able to see because it would be great to be out to see a live chart or a dashboard or something that sort of shows what versions are out there, which ones we need to be aware of that are still there and even what users are using and what versions they can maybe even move them on, if we need to. But there's definitely a bunch of things that aren't immediately obvious. And I don't know how many people actually think about this early on, but it's critical to actually think about it early on.
MATTIA: Yeah. I was going to share what we do which is very similar to what Aaron said just maybe for the listeners to have some context. The first thing you can do is basically what Gmail does, which is every time a web app sends a request, an [inaudible], it will send the version of the app with the request and the backend can check. And the backend can check if you are sending a request from the same version that is the most recent deployed version. And if it's not, this sends back a header and the same way that Gmail does, it will display a pop up that is like, "Hey, we have a new version if you want reload." And on top of that [inaudible] is that we have a dead man's switch. So if we accidentally deploy a broken version, a part of this process, the frontend application tracks in the headers. And if a special header is sent, it force reloads, which is not nice for the user but it's better and sometimes is critical to do so.
CHARLES: Right. I remember that was that case where Gmail released something. They were doing something with broken service workers and the app got completely and totally borked. I remember that my Twitter blew up, I don't know, about a year ago I think, and one of the problems I don't think they had was they did not have that capability.
MATTIA: I mean, you learned this the hard way sadly. But I think these two things are definitely crucial. And the third one, [inaudible]. I thought, Luke, you had the ability that Aaron was talking about like tracking versions in the world. And I think that's more useful for stats so that you know how often your users update. And then you can make the design decisions based on that and based on how much you want to support in the past.
CHARLES: Yeah. Crazy thought here but it almost makes me wonder if there's something to learn from the Erlang community because this is kind of a similar problem. They solved 20 years ago where you have these very, very long running processes. Some of them there's some telephone servers in Sweden. They've been running for over a decade without the process ever coming down. And yet they're even upgrading the version of Erlang that the VM is running. And they have the capability to even upgrade a function like a recursive function as it's running. And there's just a lot of -- I don't know what the specific lessons are but I wonder if that's an area for study because if there's any community that has locked in on hot upgrade, I feel like it's that one.
AARON: Makes me feel like we got a whole bunch of work to do.
CHARLES: We welcome them. I'm always happy to give people plenty more work to do. No, but how they manage even being able to do migrations on the memory that's running. I don't know if it's something that's going to be achievable but it sounds kind of like that's the direction that we're heading.
CHARLES: I'm curious because when we're talking about this, I'm imagining the perfect evergreen app but there's also feels like there's maybe even a tension that arises because one of the core principles of good UI is you don't yank the rug out from underneath the user. They need to, at some point and we've all been there when the application does something of its own agency, that feels bad. It feels like, "Nope, this is my workspace. I need to be in control of it." The only way that something should move from one place to the other without me being involved is if it's part of some repeatable process that I kicked off. But obviously, things like upgrading the color of a button or fixing a layout bug, those are things that I'm just going to want to have happen automatically. I'm not going to worry about it. But there is this kind of a gradation of features and at what point do you say, "You know what? Upgrading needs to be something that the user explicitly requires," versus, "This is something that we're just going to push. We're going to make that decision for them."
LUKE: Yeah, it's a great question. One of the things that I'm curious what you all think is when you think about the mental model that our users have of working in a browser app, do you think that there is a mental model of, "Oh, when I refresh, I might get a new version." Do people even think about that? Or are they just like your example about a button color changing as kind of a minor thing. I don't even know if I could endure stack. We've all been I think in situations where you do a minor redesign and all of a sudden, all hell breaks loose and users are in revolt. Take the slack icon. So, I think it's a fascinating question.
CHARLES: I don't know. What's the answer? Do you always ask for an upgrade just observing? I don't have any data other than observing people around me who use web applications who don't understand how they actually work under the covers. I don't think it's the expectation that this code, this application is living and changing underneath their feet. I think the general perception is that the analogy to the desktop application where you've got the bundled binary and that's the one you're running is that's the perception.
AARON: I'd say the difference there is that, and with all these new ways of deploying, we're shipping small things faster in multiple, multiple times a day or even an hour. So it's not the sort of thing you really want time to use. There has been an update, need to upgrade as well. And that's the difference between the desktop mentality. And if that's the mentality they have, it sits quite a bit of a shift, I guess.
MATTIA: It makes me think that one of the tools that the users -- if you take a look at the general public, there's probably one tool that everybody can relate to which is Facebook. So, I think if there is a way to say what do people generally expect. There is a business user which I think we are often most familiar with but the general public, probably what they're most familiar with is what happens in Facebook. And I don't use Facebook almost. I haven't used it for a couple of years but I wonder how much of what people experience in Facebook actually impacts the expectations around how applications should behave.
LUKE: I think that's a really good question. I do think your underlying point of you have to know your own users I think is an important one also. Obviously, some folks are going to be more technical than others or some audiences will be more technical than others. But I would even question, Charles, your suggestion that people think of it kind of as like a binary that it stays the same until you refresh. I think people have an idea that web apps improve over time or sometimes they get bugs but hopefully that they improve and change over time, and that there is a tradeoff there that means sometimes there's something new to learn but at the same time, you get new features. But I don't know that people necessarily associate that with and it happens when I hit reload or it only happens when I open a new browser, like I don't know that it's that clear for people in their head.
CHARLES: Right. I can see that. But the question is if the evolution is too stark, I think people tend to get annoyed. If they're in the middle of a workflow or in the middle of a use case and something changes, then it gives it a feeling of instability and non-determinism which I think can be unsettling.
LUKE: Definitely. We all value, as engineers, we value getting into that flow state so much of like, "Oh man, I'm being productive. I don't have any distractions." And you kind of owe that to your users also to be able to let them get into that state with your app and not be throwing up, "Hey, there's new stuff. Reload." "I'm in the middle of something. Sorry."
CHARLES: Yeah. I definitely do the same thing. Sometimes, I let iOS be bugging me to upgrade for a month until I finally start to feel guilty about security and actually do the upgrade.
CHARLES: Although once they started doing it at night, it actually made it a lot better.
LUKE: That's an interesting idea, too. I think there's a natural tension between the lower integration risk that we have as the engineering teams of shipping very frequently. Aaron mentioned shipping a dozen times a day. We certainly have been there and done that as well. I would say on average, we ship a few times a day. But the reason that we do that is because we know the faster we get code into production, the faster we can trust it. So it passes all your tests, it passes your [inaudible], but you don't really know if you're being honest. You don't really know until there's thousands of people using it in production. And yet, this conversation makes me think about there is a tension between how frequently you do that versus your users' kind of comfort level and expectations.
CHARLES: And maybe there is a thing where you can kind of analyze on a per user basis how often they're active in the application and try to push updates on times that are customized to them.
LUKE: Like when a user has been idle for 30 minutes or something like that.
CHARLES: Yep. Or even like track trends over months and see when they're most likely to be idle and schedule it for them.
LUKE: Good point.
CHARLES: Something like that.
TARAS: I have an idea. We should introduce screen savers into web apps. And so when the user stops using the app, just turn on screen saver and do the upgrade.
LUKE: I can see the VC patch. It's after dark but for the web.
CHARLES: Enter install flying toaster.
AARON: It does that to open up the idea as well of that automated checking that things are okay well after the fact because it's all right to sit there maybe activate something and sit there even for an hour and make sure there's no bug request coming in. But if no one's actually received your app, then of course it's not going to come in. And it's very easy to kind of move onto the next thing and forget about that. I guess it's not something that I've ever put a lot of time in and I haven't worked anywhere that's had really great automated checking to see is everything still okay. And I guess it's an interesting thing to start thinking about as well an important thing.
CHARLES: Like actually bundling in diagnostics in with your application to get really fine grained information about kind of status and availability inside the actual app.
AARON: I'm not exactly sure, really. I guess I maybe wasn't thinking about inside the app but I don't know what it looks like exactly. But there is that element of shipping fast and getting stuff out there. But are we really making sure it all works later on when everyone's actually using it.
CHARLES: Do you only use just [inaudible] or do you assign like a build number or using SemVer? What's a good strategy?
LUKE: In our case, we use git tags. And so, our CI deployment process for our Ember apps basically looks like this is we work on a PR, we'll merge it to master. If it builds, the master gets deployed into our QA environment automatically using ember-cli-deploy from our CI server. And then once we're happy with how things are on QA, we do git tag or actually I use an ember addon called ember release that does that tagging for me and I'll tag it either in patch minor or major, roughly [inaudible] although it doesn't matter that much in the case of apps. When there is a new tag that builds green on CI, that gets deployed automatically into production by ember-cli-deploy. And so, that's kind of a basic flow. That tagging, just to be clear, the SemVer tag is just going to be number.number.number. You can get more sophisticated than that and I think both Aaron and Mattia have a system where even in the PR stage, there's automatic deployments happening. So maybe one of you want to mention that.
AARON: We're slightly different. Every time we push the pull request, that gets deployed to production. We're able to preview up pull requests in production before we even merge into master which we find super useful to send out links to stakeholders and maybe people that have raised bugs just to get them to verify things are fixed. And then at the point that it's all Google merged, the pull request to master which will automatically do another deploy which is the thing we'll ultimately activate, we activate it manually after the fact. We just do a little bit of sanity checking but we could automatically activate that on merge to master as well. But yeah, the being able to preview a pull request in production is super powerful for us.
CHARLES: That is definitely a nice capability. It's hard. It's one of those certain workflows or patterns or tools that you remember life before them and then after them and it's very hard to go back to life before. I would definitely say kind of the whole concept of preview applications is one of those.
AARON: Absolutely. It's a daunting concept if you're not there, previewing something that's essentially a work in progress and production. And there are some things you want to be careful with, obviously. But for the most part, it's a super valuable thing. As you say, it's a world where once you're there, it's very hard to step back and not be there again.
CHARLES: So I had a question about, we talked about I think it was 162 plugins around the ember-cli-deploy community. What for you all has been the most surprising and delightful plugin to arrive that you never imagined?
MATTIA: That is a good one. I'm pulling up the list. What I can tell you, for me, it's not about a specific plugin. The surprising part was the sheer amount of different strategies that people use for the shipping part. At least, I found that the build part is similar for most people, like most people want to do the things that you're supposed to do. So, you want to build your application and then you want a minify it in some way. And there's a bunch of options there from gzip to more recent technologies. But the way people deliver it to servers and the difference in the solutions, that I think for me has been the biggest thing, where people that ship [inaudible] are people that ship directly to Fastly, people that use FTP files, people that use old FTP, people that use our sync, people that do it over SSH. We have people that ship stuff directly to a database because some databases actually have great support for large files. So we even use it as a storage. We have people that do it in MySequel, people that do it in [inaudible].
CHARLES: It's actually storing the build artifacts inside of a database?
MATTIA: Yes, I've seen them in that. It's kind of interesting like the solutions that people ended up using. And so for me, I think that that's been the most fascinating part. Because as we were saying at the beginning, I'm just seeing now, we even have one for ZooKeeper. I don't have an idea what this does but it's probably related to some sort of registration around the seven-day index. That, to me, I think has been the biggest surprise. Everybody ends up working in a different environment. And so, that flexibility that users need has been by far the most surprising one.
AARON: I think that's also been one of the challenging things, one of the enlightening things for me. I think in the Ember ecosystem, addons and even just Ember itself, it's all about convention of configuration and doing a lot of the stuff that you do for you. I think people expect them to see a lot of point to do the same thing. But the key thing here is it just really automates all the things you would do manually and you need to understand exactly what you want your deployment strategy to be before use them to see a lot of [inaudible] could do for you. You need to decide do I want to install my assets on a CDN and do I want to install my index in Redis or in console or in S3 bucket. You need to know all these things and have decided on all these things and then ember-cli-deploy will make that really easy for you. And this is one of the educational things, I think we still haven't even nailed because there are always people that want to know why this doesn't work. But deployments are complex thing and as what you were saying Mattia, there are so many different variables and variations on doing this that there's no sensible configuration ember-cli-deploy could really provide out of the box, I guess. And so, that's why we ended up with a pipeline that gives you the tools to be flexible enough to support your strategy.
LUKE: I think the closest that we come to the convention is that if any app is using ember-cli-deploy, you can run ember deployment targets or ember deploy prod and ember deploy QA and you can expect that that's going to work. What you don't know is how has it happened to be configured in this project.
Charles, your question about kind of the most surprising thing that's come out of the ecosystem. For me, I would say -- Mattia mentioned plugin packs earlier which are groups of plugins that kind work together well. And so, we've seen some plugin packs like you might expect, like an AWS pack for deploying to AWS. But the more interesting ones to me that we've seen a lot of, companies open source their plugin packs. So what you naturally fall into as a company that's adopting ember-cli-deploy that has multiple ember apps is that you are going to develop your own plugin pack for internal use because generally speaking, companies follow the same deployment pattern for each of their apps. There's usually not any reason to vary that. So then the new thing that happen on top of that is people said, "Why don't I make this open source so other people can kind of see how we do it?" And that's been a really delightful part of the process to kind of get a peek into how other organizations are orchestrating their deployments. And if people are curious about kind of looking at that themselves, you can go on NPM and look for keyword ember-cli-deploy-plugin-pack and pull up all of those. And you can kind of poke around and see what different companies have open source there.
CHARLES: I actually love all three of those answers because it really is for me when you have a constellation of people around a particular problem, it's the surprising solutions that emerge that are some of the most exciting that would have lain hidden otherwise. It would have been kind of buried beneath the source of company A or company B or company C but actually having it all out in the open so that you can inspect it and say, "Wow, where has this solution been all my life?" Something that you have never imagined yourself.
LUKE: That's a great question. Aaron or Mattia, one of you guys wanna talk a little bit about the history here?
MATTIA: Yeah, sure. We've been thinking about this as well. In the past, a guy on the ember-cli-deploy team, Pepin, has actually started this effort and it kind of prototyped this very idea of separating the ember-cli-deploy part from Ember CLI itself and make it a bit more generic. And he started a project called Deploy JS which I can give you a link for the show notes later. I don't think that the project is currently maintained but definitely the start of the effort is there. And the funny thing is that it was surprisingly easy. I think that we didn't get there mostly because we just all use Ember at work. As you know, open source is mostly motivated by the needs that an individual or a group of people have. But if any of the listeners were very interested in this, I think they should definitely get in touch and we will be happy to talk to them and see what can be done here.
CHARLES: I think the wider community could definitely benefit from that because most of the blog posts and talks I've seen that concern themselves with deployment of single page applications, it's still much more of the tutorial phase. Like, "This is how I achieve getting this deployment strategy." Not, "This is how I repeatedly encode it as a program," and leverage it that way. And so, definitely getting that message out to a wider audience, I think it's a -- what's the word? It's an underserved market.
LUKE: Yeah. I really like this idea also. I think about this ImmutableWebApps.org, if you look at it. It's sort of a manifesto with conceptual description of what are the qualities that your app has to have to qualify as an immutable web app which I think is kind of a funny idea but one that people can start to get their heads around and compare that description to their own apps and say how do I hit this or fall short. And it reminds me a lot of kind of the idea of Twelve-Factor App which is an idea that I think came out of Heroku originally. And it's an idea of a backend app that is portable to be able to easily move across different hosts and easily be scalable to different instances of [inaudible] if your app obeys all of these things then it's going to do well under those circumstances, that will satisfy those needs. So, I think it's a great way of thinking and probably maybe even a better entree into the conversation with the wider community than a library, this is certainly a library called ember-cli-deploy.
MATTIA: Thank you so much. It was a great time.
AARON: It was a pleasure. Thanks for having us.
LUKE: Thanks so much.
CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at email@example.com. Thanks and see you next time.