Terraform + Fastly: now better, together | Altitude NYC 2019

Sweeping updates have been made that reconcile Fastly's Terraform provider with key Fastly app functionality. Davy Stevenson, Sr. Director of Application Engineering, offers a glimpse into the latest updates, plus a future roadmap for ongoing Terraform upgrades.


Hello everyone. So I'm Davy Stevenson, I am responsible for Fastly's customer-facing UIs and APIs. So I would love to hear from you your thoughts and suggestions on how Fastly could continue to improve our customer-facing interfaces. I live in Portland, which you might already know because I'm wearing plaid, and I'm really happy to be here today. So I am going to be asking some questions throughout this talk. So we're going to practice getting interactive here. The first question is a really easy one because you should all answer yes and if you don't, I'll have questions which is, who here automates the deployment of their software onto their production servers? Yay. Who here automates their deployment of their Fastly services? Okay, we got a bunch of hands too. This is great. Well not necessarily everyone, maybe not everyone has fully automated the deployment of their Fastly services. And why should your edge network be any different than your main production services?


So the problem with manual deployment is one, your deployment is manual and that can lead to configuration drift. No matter how good your processes are written up for how to do deployments, humans are fallible and mistakes will be made. And this means that a lot of these processes become not reproducible and it also can become really, really hard to see the outcomes ahead of time of your deployment. It also makes rollback really hard because the same problems that are going to lead to inconsistencies in your deployment will then apply to your rollback as well. And oftentimes, this also means that your changes can't be traced.


So what is our goal instead? You know, we want Fastly services to be easily reproduced, we want them to be consistent. We want to know exactly what state they're in and we want the state to be what we want them to be. We want the processes to be repeatable and predictable when we're making changes to our Fastly services. It would be really, really great if the configuration was also continuously tested, and if we had visibility and actionability into the process of deploying changes to your Fastly services. So how do we do this? So best practices today are talking about declaring your service configuration as code. This now means that your configuration can be stored in version control like other software. This means that you can define CI and CD pipelines for this configuration code, just like any other code, and those changes that you're making can be tested and continually integrated and then you can create an automated deployment pipeline.


So this is where Terraform by HashiCorp comes in. Terraform is one potential choice that you could use for a software solution for this problem. And their tagline, of course, deliver infrastructure as code. So now Terraform becomes the glue between your configuration that's stored in GitHub and actually getting that configuration deployed out to Fastly. So this talk, at the core level, is a love story talk. Like any love story, the path has not always been easy and there have been challenges to overcome along the way. I'm also freely admitting that I'm anthropomorphizing software for entertainment purposes so we're going to go and run with this analogy.


We're going to start by talking about HashiCorp. So the Fastly Terraform provider was one of the very first Terraform providers and it was written originally by HashiCorp. releases.hashicorp.com is fronted by Fastly, which means that HashiCorp wrote the Terraform provider for Fastly to Terraform, the Terraform website for downloading Terraform. And so this initial spark went both ways and, in fact, Fastly provides free services for Terraform under our open source program as well.


So for many of our customers, Fastly has become an integral part of their production infrastructure. Being able to consistently, reliably and safely roll out to changes to Fastly is really critical. And many customers choose Terraform to perform this action. And we know this because if you check our raw count of API requests that come from the Terraform provider, we're seeing a nice consistent growth over time. So more and more customers are using Terraform or they're just hitting Terraform more and more often, we don't know. So another quick poll, here's the interaction part. How many people here are using Terraform right now? Well, I see a bunch of hands. How many people are interested in using Terraform? Couple more. Excellent. You're in the right spot.


So the initial spark between Terraform and Fastly was there. Both sides could see the potential that the other provided but as I alluded to earlier, it is rare for a love story to be 100% smooth and as we all know, good relationships take energy and effort and recently the relationship between Terraform and Fastly has gone through a bit of a rough patch. As it was mentioned earlier, the Terraform provider was originally written by HashiCorp and while Fastly has occasionally contributed, our contributions have been spotty in the past, and this has led to not the consistent level of support as it turns out was necessary.


At the end of the day, our customers that we're using Terraform weren't always feeling fully supported by Fastly and we weren't living up to the high level of customer support that our customers have grown to expect from Fastly as well. So the Terraform provider didn't support all of Fastly's features. There were bugs that customers were encountering. As I alluded to earlier, we weren't providing a lot of dedicated support so sometimes we'd be contributing and helping out and sometimes people would ask for things and there would just be crickets. Customers would have to extend the Fastly Terraform provider themselves to add needed features and all of this was kind of leading to some general inconsistency.


So that wasn't a really... we didn't really feel good about this, right? This is not the place that we want it to be. And so we heard you, we listened and in 2019 we did something about it. So in the beginning of 2019, we made it a goal to spend the time and energy to work on improving this relationship between Terraform and Fastly to continue to drag my analogy out. Relationship building requires effort and it also requires time. But by spending that time and energy, it was important to Fastly because Terraform was important to you, our customers.


So what did we do in 2019? We made some significant UX improvements, in particular, the Fastly or the Terraform diff view is a lot easier to use now. We've added all the versionless resources, versionless snippets, dictionaries, and ACLs, we've added support for some new logging endpoints, like Azure and Splunk, and we fixed a bunch of bugs.


So we're going to go into some code examples here. So this is an example for dynamic snippets. Inside the Fastly service view one resource, we have a new property called dynamic snippet where we can define the name, type, and priority of the dynamic snippet. Let's pay attention to the name here because we're going to come back and use that later. So there's a new top-level resource called Fastly service dynamic snippet content V1 and the first two lines there are kind of boilerplate. You'll see these in the rest of the examples as well. So the first one is just grabbing the service ID of that earlier service and the second one is iterating through all the potential dynamic snippets to find on that service to find the one with the name router. So now we have the service ID and the snippet ID in Fastly of those two of that snippet.


And then we simply have the content property on that dynamics snippet and we just put the content in there. In this case, it's for rewriting URLs in VCL. So now I'll go through an example for dictionaries. So this is another URL rewriting example and in this case, we're doing something a little bit different. We're going to store the data inside a variable which we're giving the name URL rewrite dictionary. Inside this variable, we're saying this is going to be an object with a name which is a string and a map of strings that we're going to call items and we're going to set the name to URL rewrite dictionary and we're going to set the items to the URLs we want to rewrite. Very similarly in the Fastly service V1 example resource, here we're going to have a new dictionary property where we can set the name to the variable that we defined earlier.


And here we have the new top-level resource, Fastly service dictionary items v1, same to boilerplate code to get the service ID and the dictionary ID of that object. And here we have the items property and we're just simply going to set the items from the variable we listed earlier into this items property here. But this could come from other things, it doesn't have to come from a variable like this. Those come from other objects, other Terraform definitions that you already have in place.


So this is the longest example for ACLs. The first part, very, very similar, right? We're setting that variable for office IPs. This is an example where we're setting an ACL, there's going to be a service where we only want employees to access to it and they only get access from the office IP addresses. So very, very similar. We have the name, which is a string and entries, which is a list of strings here. Set the content there, same thing with the Fastly service. We have a new ACL property, which we can define and link to the name of that ACL.


And here again, new top-level resource, Fastly service ACL entries v1, same to top boilerplate code to get the service ID in the ACL ID. And here I'm going to — there's a little bit more involved. For example, we're going to be using the dynamic blocks syntax, this was added to Terraform v12. So this allows for expressions to actually be evaluated inside the block here, which gives you some extra power. So in this case we're iterating over those lists of IPS and gathering them from the variable we defined earlier and then setting the content block for that entry to be the IP address and then just a constant sub-net and negated value. But here again, with this dynamic property, you could do more complex things.


So you could be getting these IP addresses from different Terraform provider definitions that were already evaluated somewhere else, perhaps from your Google cloud provider interfaces or your AWS interfaces. Maybe you're getting your IP addresses from a load balancer or something like that. The dynamic block also allows you to do things like combine things together if you're getting two lists from two different places.


So that's all the code that I'm going to show. So as I said earlier, we added some new logging endpoints. So this is the full set of logging endpoints that we support in Terraform right now. We added the Azure Blob Storage and Splunk. We are going to continue adding some more logging endpoints to Terraform over the next year. If there's a logging provider that you use or you want to use with Terraform, please let us know so that we can do a good job of prioritizing which ones we add next.


So another big change that happened this past year was Fastly taking over ownership and support of the go-fastly client library. This is a client library that the Terraform provider is built upon. So all of the features that have been added to Terraform have therefore also obviously been added to the go-fastly client library. I want to give a special thanks to Seth Fargo for owning and supporting this client library for the many years before now and for really making the facilitation of transferring ownership over to the Fastly org really painless. So you can see here, go-fastly is now located under the Fastly org within GitHub. So that was kind of what we did in 2019 but we're not done yet. We have a bunch of work lined up for 2020 as well that we're excited about.


So as Rajib and Kimmie mentioned earlier, the big thing is that we are working on adding WAF support to the Terraform provider. This has been another top request from customers for a long time. The project has already been kicked off. Stay tuned for more information on when that actually is going to land. If you are a customer who is using our WAF product and want to use Terraform to manage the WAF, let us know so that we can make sure you get updates as quickly as we can get them to you.


So we're also going to be forming a new dedicated team in 2020 to support the Fastly Terraform provider, the go-fastly client library, and also the Fastly CLI tool that was demoed earlier today around the Compute@Edge project. So all of those libraries, all those projects are written in Go. So if you know anyone who's interested in working in Go and want to do this open-source CLI-type work, send them my way, I would love to talk to them. So we also are just overall increasing our commitment to ensuring that new Fastly features are included into Terraform in a timely manner so that we can continue to keep this relationship between Terraform and Fastly strong.


So I have some extra time. So I'm going to talk about something that I think is really cool, which is GitHub Actions. So GitHub has recently released GitHub Actions into beta and I'm excited about the potential this provides. Who here has done anything to play around with GitHub Actions thus far? Excellent, I'm seeing some hands come up. So GitHub Actions is a way to be able to spin up containers and run code on GitHub's own infrastructure and so you can use that to build out CI/CD.


So quick overview. Continuous integration is the idea that you want to validate configuration as early in the process as possible. Ideally, when PRs are opened or even when code is pushed to GitHub the first time. This gives you early alerting for formatting and configuration problems. You can run linters, you can get early warnings if you need to fix things before you know — just let the computers tell you what you're supposed to be doing. And then continuous delivery is the idea that you can completely automate the deployment of configuration when PRs are merged and that allows you to then fully automate as much of the deployment pipeline as you might need to.


So I'm going to talk a little bit about how we create Fastly with Terraform and Fastly. So we have used Terraform to manage a number of our own services for a long time. This includes the Fastly UI at manage.fastly.com, our API at api.fastly.com, and our assets URL at assets.fastly.com. Each one of these projects has a production and a staging service that we use to test out changes and we're currently exploring how we might use GitHub Actions to better automate the CI/CD for these projects.


So HashiCorp has already released Terraform GitHub Actions which is great. It means you can easily get access to being able to run Terraform format in it, validate, plan and apply really easily in GitHub Actions. So here's a little overview of the structure that we're exploring for using GitHub Actions for the CI part of the pipeline. So here we have Git Push, pushing that code to GitHub. That's going to trigger a GitHub Actions, we're integrating in the Terraform GitHub Actions into here. As I said before, we have the staging and production service and so we can run format and It Valley and plan on each of those. You can choose yourself if you want to run these in one serial action or if you want to run four in parallel. There's pros and cons to both. We're right now exploring running them in four in parallel so that if you get errors in two, you can get those errors at the same time instead of fixing one running and getting another error and having to fix it again.


As we would expect, GitHub Actions, allows you to send out notifications to Slack or your other favorite chat program of choice. And one cool thing that it also does is has the ability to add comments back on the PRs for the outcome of these actions, which is really, really cool and I'll show some examples of what that looks like. And then of course, since we always push code perfectly the first time, then we get that green checkmark and we're ready to go. So here's some screenshots of what this looks like. So here we can see all these actions that we've defined here. There's eight of them defined four for each of production and staging. Can't even see them all in this one, but you can see that they've all successfully run, and this PR's waiting for a reviewer so it can be merged.


So what do some of these comments look like? So here's an example of what we might get if Terraform format fails. So in this very contrived example of adding a new domain for www.altitude-newyork.com, we're going to see here that we screwed up the spacing, didn't get the format right and it's telling us right there with syntax highlighting what needs to be changed in order to make format happy again. And here, this is right on the main PR right? You don't have to dig around, you don't have to go to a different tab, you don't have to scan through thousands of log lines. It's just right there and you can see it.


So same thing here. We have Terraform validate, it's unhappy and it tells you exactly what line it's on and tells you exactly what you need to do. So we're missing a required argument. It's saying, "Hey, your service needs a name. It wasn't found for this Fastly service v1 API service product. You better go add a name there. Here's your line number." Again, you don't have to dig around in any other tabs. Also, it's useful when things are successful. So here the Terraform plan is successful and this now provides you a really good visual indicator of what actually is going to change if you merge that PR. You know immediately hear that, "Ah, if I merge this PR, we're going to add a new domain with a new apex domain for altitude-newyork.com in addition to my www domain." It's really easy to see that right on the front page of the PR, you don't have to dig around in the code and try and figure out what changed.


And so that leads us to what could be a potential CD pipeline. So this is, again, just one type of architecture that you could choose for using CD for Terraform. So here we are, we like our PR, we merge that PR, that triggers a GitHub Action. And again, we're going to use the Terraform GitHub Action commands and it's, again, we have staging and production, so we're going to kick off two. For staging, I just want Terraform apply and I immediately want to activate that version so we have something on staging to test within minutes. And then that outputs a message to Slack or your other favorite chat program of choice.


For production, we might want to do something a little bit different. I don't want to just automatically activate it, there might be some other checks and balances that we have. We might want to actually go test this stuff on staging before activating in production, right? So instead we set Terraform apply to only create a new draft version and that still gives us the ability to have a human being go into the UI, double-check, do a diff, test in staging, verify everything's right and then all they have to d0 — the manual processes is now stripped down to simply pressing the activate button in the UI to make the active version.


So like I said, you can have these things output to Slack. So, and this can make it really, really easy to provide the tools that you need to make this happen. So here you can see this was the output of a push event. So that's the CD pipeline and it's saying, "Hey, Terraform created version 11 of this service. It's waiting to be reviewed and activated." And it gives you a link right to the URL in Fastly so that you can actually go to that version of the service, check it out, run the diff, activate.


So let's go back to the very beginning where we were talking about what the goal of a good automated deployment for Fastly might look like and let's look back and see if GitHub and Terraform and Fastly now together can meet all these goals. Services can be easily reproduced. Yes, I think Terraform can easily allow you to meet that goal. Services are consistent. Yes, Terraform provides that consistent application of configuration. You can define processes that are repeatable and predictable, yes. Configuration is continually tested, yes. You can use things like GitHub Actions or any other CI platform for that and now it provides you additional visibility and actionability that you may not have had before. So great success. I think we've met all of our goals.


So I ask you to give it a try. We have fully-featured docs on Terraform.io for the Fastly provider with lots of examples and of course, reach out to us if you have any other feature requests. I'd love to hear from you. So watch this space, I'm excited to see what happens with Fastly and Terraform in 2020 and I hope you are too. Thank you very much.