Webhooks are great, so many services now support them but I found actually doing anything with them a pain as there are no standards for what goes in them and any 3rd party service you wish to integrate with has to support the particular hooks you are producing.
For instance I want to use SignalFX for my metrics and events but they have very few integrations. A translator could take an incoming hook and turn it into a SignalFX event and pass it onward.
For a long time I’ve wanted to build a translator but never got around to doing it because I did not feel like self hosting it and write a whole bunch of supporting infrastructure. With the release of AWS API Gateway this has become quite easy and really convenient as there are no infrastructure or instances to manage.
I’ll show a bit of a walk through on how I built a translator that sends events to Signal FX. Note I do not do any kind of queueing or retrying on the gateway at present so it’s lossy and best efforts.
AWS Lambda runs stateless functions on demand. At launch it only supported ingesting their own Events but the recently launched API Gateway lets you front it using a REST API of your own design and this made it a lot easier.
For the rest of this post I assume you’re over the basic hurdles of signing up for AWS and are already familiar with the basics, so some stuff will be skipped but it’s not really that complex to get going.
The Code
To get going you need some JS code to handle the translation, here’s a naive method to convert a GitHub push notification into a SignalFX event:
This will be the meat of the of the processing and it includes a bit of code to create a request using the https module which includes the SignalFX authentication header.
Note this creates dimensions to the event that is being sent, I guess you can think of them like some kind of key=val tags for the event. In the Signal FX UI I can select events like this:
And any other added dimension can be used too, events shows up as little diamonds on graphs, so if I am graphing a service using these dimensions I can pick out events that relate to the branches and repositories that influence the data.
This is called as below:
There’s some stuff not shown here for brevity, it’s all in GitHub. The entry point here is handleGitHubPushNotifications, this is the Lambda function that will be run. I can put many different ones in here and in the previous code and share this same zip file across many functions. All I have to do is tell Lambda to run handleGitHubPushNotifications or handleOpsGeniePushNotifications etc. so this is a library of functions. See the next section for how.
Setting up the Lambda functions
We have to create a Lambda function, for now I’ll use the console but you can use terraform for this it helps quite a lot.
As this repo is made up of a few files your only option is to zip it up. You’ll have to clone it and make your own config.js based on the sample prior to creating the zip file.
Once you have it just create a Lambda function which I’ll call gitHubToSFX and choose your zip file as source. While setting it up you have to supply a handler. This is how Lambda finds your function to call.
In my case I specify index.handleGitHubPushNotifications – uses the handleGitHubPushNotifications function found in index.js.
It ends up looking like this:
Once created you can test it right there if you have a sample GitHub commit message.
The REST End Point
Now we need to create somewhere for GitHub to send the POST request to. Gateway works with resources and methods. A resource is something like /github-hook and a method is POST.
I’ve created the resource and method, and told it to call the Lambda function here:
You have to deploy your API – just hit the big Deploy API button and follow the steps, you can create stages like development, staging, production and deploy API’s through such a life cycle. I just went straight to prod.
Once deployed it gives you a URL like https://12344xnb.execute-api.eu-west-1.amazonaws.com/prod and your GitHub hook would be configured to hit https://12344xnb.execute-api.eu-west-1.amazonaws.com/prod/github-hook .
Conclusion
That’s about it, once you’ve configured GitHub you’ll start seeing events flow through.
Both Lambda and API Gateway can write logs to Cloud Watch and from the JS side you can see do something like console.log(“hello”) and this will show up in the Cloud Watch logs to help with debugging.
I hope to start gathering a lot of translations like these and am still learning Node, so not really sure yet how to make packages or classes but so far this seems really easy to use.
Cost wise it’s really cheap. You’d pay $3.50 per million API calls received on the Gateway and $0.09/GB for the transfer costs, but given the nature of these events this will be negligible. Lambda is free for the first 1 million requests and you’ll pay some tiny amount for the time used. They are both eligible for the free tier too in case you’re new to AWS.
There are many advantages to this approach:
- It’s very cheap as there are no instances to run, just the requests
- Adding webhooks to many services is a clickfest hell. This gives me a API that I can change the underlying logic of without updating GitHub etc
- Today I use SignalFX but it’s event feature is pretty limited, I can move all the events elsewhere on the backend without any API changes
- I can use my own domain and SSL certs
- As the REST API is pretty trivial I can later move it in-house if I need, again without changing any 3rd parties – assuming I set up my own domain
I have 2 outstanding issues to address:
- How to secure it, API Gateway supports headers as tokens but this is not something webhooks tend to support
- Monitoring it, I do not want to some webhook sender to get in a loop and send 100s of thousands of requests without it going unnoticed
