Updated: Updated: Architecture
It’s been two months since I blogged last on this site. That’s because I took the Architect Elevator in two directions at the same time (yes, IT can do some things that physics can’t): first, I was up in the penthouse authoring posts for the AWS Enterprise Strategy Team Blog (please have a look) and second I was visiting the cloud integration engine room . Here’s what I brought back with me.
The Architect Elevator is all about connecting levels: levels of abstraction, levels of corporate hierarchy, levels of strategy. To be effective at connecting these levels, it’s important for architects to actually visit them every once in a while. This can mean a physical (or video-virtual) visit to the penthouse for a strategy or steering meeting, a self-guided tour of reading the latest analyst reports, or trying out some technologies down in the engine room. In any case, it’s important to actual get out of the elevator lest you be rendered a “lift boy/person”, as warned in my book.
My generally recommended way for visiting the engine room is to pair with a developer on a real project. In a day or two you’ll learn as much as you would over three months of architecture reviews and steering committee meetings. Other times, it’s good to venture out on your own and try to build something. It’ll be more time-consuming but you’ll likely encounter many nuances and stumbling blocks that an experienced developer would just glance over with “just click here”.
The Integration Engine Room: Lots of Machinery
When visiting the engine room, it’s allowed and in fact encouraged to give yourself a head start by picking a place that you have some familiarity with. Having a lot of technical domain knowledge but little experience with the respective tools maximizes the number of naive questions with insightful answers you will be asking.
My area of technical domain knowledge is surely integration. Having co-authored Enterprise Integration Patterns in 2003, messaging architectures have stayed close to my heart for almost two decades. The recent popularity of distributed systems in form of microservices architectures and serverless solutions gave integration patterns an unexpected boost, making integration no longer an afterthought but an integral part of the application design. In an area with few “right” answers, patterns that lay out decision options and trade-offs have proven to be a valuable design tool.
Not surprising, then, many new integration tools have arrived on top of the existing stable of Enterprise Service Buses. Virtually every cloud provider includes a message queue in their service offering, ranging from Amazon SQS and SNS to Google Pub/Sub and Azure Storage Queues, augmented by hosted versions of popular messaging frameworks like Amazon MQ and Azure Service Bus. On top of this, we see a flurry of event-related tools like Amazon EventBridge, or Azure Event Grid. So, there’s no shortage of integration-related tools in the cloud, some might even consider it an oversupply as pretty much every vendor has built a “which tool to choose when” page.
The Venerable Loan Broker
I find that I can improve my focus in the engine room if I equip myself with a concrete task. This doesn’t have to be a production system by any means but rather some concrete goal that helps me overcome the inevitable hurdles I will face. Also, a concrete objective counteracts the tendency to just build whatever happens to be easiest on the particular tool–you’ll find plenty of those examples in the vendor tutorials. So, picking your own goal provides you with an unbiased rating of what’s easy and what’s hard with a given tool set.
For this engine room visit, I dug out the running example from Enterprise Integration Patterns, the Loan Broker example:
It’s a fictitious example that draws on a well-known business domain (customers requesting home mortgage quotes) that is simple enough to immediately understand but nevertheless includes enough nuances to challenge common tooling. The key tasks of the example are straightforward:
- A Customer submits a loan application with personal data and desired terms, such as loan amount and duration.
- The Loan Broker enriches the request with the customer’s credit score retrieved from the Credit Bureau.
- The Loan Broker submits the application to multiple Banks, for example through a Publish-Subscribe Channel.
- The Banks reply with a loan offer if they are willing to service the loan.
- The Loan Broker aggregates the results, for example by selecting the best offer or by simply concatenating them.
- The Loan Broker returns the combined result to the Customer.
Staying close to home, I chose to implement this application on AWS’ integration cloud services. You can read the whole experience in a blog-mini-series on my site eaipatterns.com. Documenting the engine room deep-dive in three posts of about 4000 words each might hint that there was quite a lot to be learned from actually implementing an example. Admittedly, I didn’t take any training and just went for it–I believe it’s best to come into the engine room unencumbered.
Come Back With Dirty Hands and a Full Head!
Diving into the engine room is only useful if a) you actually get your hands dirty and b) if you bring some learnings back to the other levels. After all, the architect elevator is a connecting element across the levels. I perhaps learned more during this engine room visit than I had bargained for, but I was happy to return back with a slew of fresh insights:
Integration Development Has Come a Long Way
In the old days, you’d consider your first milestone the installation and configuration of software packages. I remember working on integration software being a valid reason to get your laptop upgraded to a beefier model. No more such luck: “serverless” solutions are real and leave you with nothing to install, provision, manage, or scale. You are up and running in minutes in an environment that can easily scale when you need it to.
Low-code, Polyglot Complexity
You will write a lot less code. My solution primarily consists of code snippets containing 30 lines or less because serverless solutions have all but eliminated the “skeleton code”, all the stuff that’s needed just to invoke a method or write to a database.
However, the code you’ll write likely uses many different languages. You’ll be writing Lambda functions in Python or JavaScript, develop the workflow in the Amazon States Language (although there’s a pretty nice visual editor for it), enter expressions in JsonPath, and filter command line outputs in JMESPath. For old dogs like me where everything used to be in Java, this turned out to be quite a challenge.
Naturally, each language and representation has its corner cases. For example, JsonPath tools differ in their handling of lists, which I attribute to XML/XPATH, from which JsonPath derives, not having any lists. You know there’s something going on when a popular JsonPath on-line evaluator shows results from four different implementations.
Just for kicks, enter the following JSON fragment–it’s a valid list of two objects in an array:
[
{"a": 1},
{"a": 2}
]
and apply the XPath expression $[0] against it - you’ll see quite different results from different implementations. Now how do you explain to upper levels that your project is delayed because your Lambda function returned a JSON list which isn’t easily parsed by your orchestrator’s JsonPath implementation? It might be easier to just say “we hit some snags”. Riding the elevator puts you in touch with reality but sometimes reality bites.
Abstractions Are Great Until Something Breaks
It always amuses me when I encounter the exact problems I have written down before and am once again stumped by them. I concluded a good while ago that failure doesn’t respect abstraction and the same holds true in the cloud integration engine room. Soon enough, I had many tabs of CloudWatch events logs open to debug issues. So, appreciate the abstractions because they make routine tasks easier but be prepared to turn off the autopilot when a warning light goes on.
Security is a First-Class Citizen in the Cloud
When we built monolithic software, security settings were largely an affair of creating Linux users, setting database passwords, and configuring network routing. Fine-grained serverless applications have changed this a lot. Each resource is access-controlled via users, roles, and security profiles. This can be annoying to new developers (who might be tempted to just “*” everything, meaning giving unrestricted access), but it gives teams a slew of new tools to control access. For an architect, this means that you’ll design security roles along with your software components. Call it “ArchSecOps” if you like…
GUI is Silver, Automation is Gold
Graphical developer tools can dramatically flatten the learning curve. If you’re not quite sure, you can browse the menus or the available icons and quickly try something out. The main risk of such tools, though, is no one being able to reconstruct the exact steps that were performed. That’s why automated deployments are king in the cloud: they give you repeatability, scalability (by deploying many instances), and maintainability. Alas, they often cause slow feedback cycles: when you’re debugging an issue, it’s quicker to try something in the GUI than developing a new automation script.
My brain is only so big, so I stuck with GUI tools and command-line until I had the solution working and then converted it into automation scripts. The risk is that you’ll forget to include something, so the only real test is to redeploy from a blank slate. It might seem tedious but the feeling that you can always recreate a full setup is well worth it.
Make Dependencies Visible
Distributed systems are difficult to grok and the more loosely coupled they become, the more hidden dependencies can become. Connections between components can occur implicitly, e.g. by components publishing and subscribing to a message channel of the same name. That name might be buried in some configuration file, making this a hidden dependency, one that’s critical for the system’s function but one that’s not easily seen in the source code.
An elegant way to make these dependencies visible is through automation: both components could reference the same message channel. Other cases, e.g. where you interact with existing systems, don’t make it quite so easy. Therefore, be sure to make these hidden dependencies visible, either in code or in documentation. Yes, this might involve drawing a picture!
Architecture (and Architects!) Are As Important Than Ever
Some folks believe that cloud platforms make many architecture decisions for you and what’s left is just a matter of product selection. Marketing diagrams sporting many prominent product icons (square, hexagon, free-form?) might support this notion. You might know that I have strong opinions about product selection not being architecture - architects are the chefs, not the folks picking tomatoes at the supermarket. Modern cloud solutions combine many elements, so there’s a lot of cooking to do!
Cloud removes many constraints that have shaped architectures of the past, such as slow provisioning and start-up times, high fixed costs, and limited transparency. However, the modern systems we design also tend to have more complex run-time behaviors that want to be well-architected. If anything, the need for architects has increased.
There’s More Than Coding in the Engine Room
There’s a frequent debate whether architects need to be able to code. Oddly, coding has a slightly different notion in integration and especially in cloud-based integration where many things can be achieved with much less code. Ironically, though, architectural considerations play a big role in such solutions as they need to consider coupling, scalability, hidden dependencies, and more.
Also, an architect might be a decent Java programmer (I considered myself “alright” back when generics where the hot new thing) but not be aware of modern deployment and run-time models such as serverless solutions and fine-grained access controls. Modern engine rooms contain a lot more than just code and libraries, so just being able to code doesn’t mean you’re in touch with the engine room.
So, to me the question isn’t so much whether architects write a lot of code. It’s more important whether they have been to the engine room recently and frequently enough to understand the available solution options and trade-offs, and a few of the dark corners. Pilots do walk-arounds of their equipment even though they sit up in the cockpit, as I highlighted in a recent blog post. So why would architects be above that useful ritual?
For reference, here the three posts from the engine room:
Make More Impact as an Architect
The Software Architect Elevator helps architects and IT professionals to take their role to the next level. By sharing the real-life journey of a chief architect, it shows how to influence organizations at the intersection of business and technology. Buy it on Amazon US, Amazon UK, Amazon Europe
