Allocating Secret Santas using an AWS Step Function workflow and every available Lambda runtime

Over the past several years, I have taken the opportunity of allocating Secret Santas for members of my family as an excuse to explore different programming languages and technologies. This year has been no different, with me opting to over-engineer the problem of allocating and notifying participants by diving into AWS Step Functions and the many runtimes available on Lambda. In this post, I wish to document how I went about designing the Step Function workflow and breaking up the problem into many specific-purpose Lambda behaviours. The final implementation can be found in this GitHub repository.

I should reiterate that this is heavily over-engineered for solving the problem at hand, with the main driver being to provide me with enough of a problem domain to explore the many different features/states (i.e. Map, Choice, Parallel) of Step Functions and runtimes available on Lambda.

Step Function Workflows?

I have had the opportunity to explore employing a Step Function workflow for several personal projects over the years, providing a high level of orchestration/durability between task/state transitions. One of the key elements I have found is knowing when to model such decisions/executions at the workflow level or leave them within the code itself. Fortunately, this problem could be broken up into several logically separate problems (parsing, allocating, notifying), which allowed me to experiment with handling failure and mapping input in parallel with choice branching.

Lambda Runtimes?

I am a big proponent of Lambda, but due to the languages and runtimes I have been exposed to in the past, I have not had the opportunity to explore many of the available runtimes Lambda has to offer. Breaking up this problem into many Lambda behaviours felt like a great opportunity to change that.

However, with the availability of the Custom Runtime API, the list of available runtimes is endless. As such, I decided to limit the scope to all six distinct supported runtimes AWS has to offer, along with a single custom runtime hosted on provided.al2. This gave me the chance to experience and implement functionality in each language runtime using more than just a simple Hello World example.

Custom Runtime

Instead of using a pre-built custom runtime, I opted to additionally take the opportunity of integrating my own personal language, which I have been developing over the year, into a Lambda context. I am sure I will be discussing my experience developing this language more in later posts, but at a high level, it is a tree-walking interpreted programming language that is targeted primarily at solving Advent of Code problems. The current working implementation is hosted on Node.js. Due to this, I was able to garner inspiration from other custom runtime bootstrap processes and how they handle the Lambda request lifecycle. I was able to package up the bootstrap into a single executable thanks to pkg and distribute it as a layer for my workflow to use.

This side project provided me with a great appreciation for the Custom Runtime API that AWS has developed, using HTTP as the common denominator for communication between the desired execution and host Lambda environment.

The Workflow

For managing and deploying the workflow, I opted to use the Serverless Framework and the de facto Serverless Step Functions plugin. This allowed me to co-locate the workflow and Lambda definitions, which I felt was very beneficial.

The resulting allocation and notification process was built up as follows:

Many of the runtimes required their own specific packaging steps, either for pulling down dependencies and/or compilation. As such, I opted to define/document these within Makefile targets, which use Docker as the primary means of providing the required execution environment to deterministically package the artefacts.

Conclusion

I really enjoyed building out this behaviour using AWS Step Functions and Lambda runtimes. In the timeframe I gave myself, I was unable to develop a sufficient local execution environment to test the workflow using Step Functions Local, but in future projects, I hope to explore this further.

The more I explored Step Functions and the available features/integrations, the more I realised that much of the behaviour I wished to achieve could be developed using high-level integrations that are already provided instead of relying solely on Lambda. For example, I could instead possibly store the allocations within S3 using the built-in AWS SDK support and send the SMS and emails via SNS and SES alike.

As the intent of this project was a combination of exploring Step Functions and Lambda runtimes, leaning on Lambda and a runtime to achieve these tasks felt right. However, in a future incarnation, perhaps the goal could be to just leverage Step Functions altogether. There is always next year… 😉