Do what I mean, not what I say And the gentle art of wrangle meaningful responses from our Generative AI tools While listening to Welcome to the machine by Shadows Fall

Bedrock picture from DuckDuckGo and not a Minecraft one

Data Governance as the Bedrock of Effective AI Governance

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

November 3, 2023

As an organisation you need a plan to address the market disruption of Generative AI. You don’t need to build your own version of ChatGPT. But you need a plan on how your organisation will deal with all the initiatives that will start. Otherwise I wish you good luck with the conversation you will have when one of your CxO comes back from some partner paid conference stating that the company will be bankrupt if you don’t invest right now.

In this series of articles I felt the need to explore some of my current thinking on where Generative AI has it’s place.

Business Braveheart

In this ever-renewing push of the newest flavour of technology, the fusion of architecture, governance, and data governance stands as the cornerstone for reliability.

As organisations navigate their discovery of the complex realm of artificial intelligence, it becomes increasingly apparent that the success of implementing one of these LLMs (ChatGPT, Bard or Bing AI) is deeply entwined with the quality, security, and integrity of the data that they need and produce.

Effective AI governance is not just about fine-tuning algorithms or optimising your LLM models. It begins with the bedrock of quality.

Garbage in, garbage out

Feedback loop

It’s the quality, accuracy, and reliability of the input data that dictates the usefulness of AI’s output. Thus, a holistic approach with a very strong foundation in data quality and it’s governance. And remember, the prompts that you use for getting results is also data that needs to be governed. How else will you establish a feedback loop on effective usage of the tool?

The Interdependence of Data Governance and AI Governance

Data governance, as I’ve stated in the previous blog posts, primarily concerns itself with the management, availability, integrity, usability, and security of an organisation’s data.

AI in any form, by its nature, operates as an extension of the data it is fed. Without a sturdy governance structure over the data that you produce, AI governance becomes a moot point. On another note I’m still surprised nobody came up with an AI that generates cute cat short clips for a Youtube channel. Wait, I’m on to something here…

Quality Data: The Lifeblood of AI

A key aspect is that the quality of data isn’t an isolated attribute but a collective responsibility of various departments within an organisation. We all know that, but where does the generated data sit?

In the past I wrote about System Thinking and I still have to plot for myself where Generative AI sits. Is it like our imagination? Where do you Master the data LLM generates for you? Can I re-generate reliable? What happens with newer generated outcomes? Are these better then the old? Is the generated response email owned by the Service Department or the AI team? These articles are as much for you as for me to fully grok where LLM and it’s outcomes sits in the system.

Security and Ethical Implications

Privacy concerns, compliance with regulations, and ethical considerations in handling and processing data are pivotal components of data governance. As AI systems often deal with sensitive information, ensuring compliance with data protection regulations and ethical use of data becomes a critical component of AI governance. The same goes for the outputs. Where are they used or stored? How do these different data providers compare? The question that popped up in my head was that within the Salesforce ecosystem we use a lot of Account data and have linked with third party providers. We enrich the data that we have of the customer with Duns & Bradstreet information or in the Netherlands with the KVK register. What happens with the ‘authority score’ if we add Generative AI in the mix? We still have a lot to discover together.

Keep it simple

Keep calm meme-o-matic

In short because I harped on it before. Organizations should:

Establish Comprehensive Data Governance Frameworks: Institute clear policies for data ownership, stewardship, and data management processes. This not only fosters quality but also ensures accountability and responsibility in data handling. Promote Cross-Functional Collaboration: Break down silos and encourage collaboration between various departments. Not just good for data quality but many more aspects in life. Leverage Automation for Data Quality Assurance: Harness the power of automation tools to identify anomalies and inconsistencies within data, ensuring high-quality inputs for AI models. Ever did a large migration from one system to another? Right, automation for the win!Continuously Monitor and Improve Data Governance: Implement systems for ongoing monitoring of data quality. We have a Dutch expression which translated goes something like “The Polluter pays”. Bad data has so many downstream effects that I almost want to advise to have a monthly blame an shame high light list. Let’s forget about that for now. I do however want to stress a carrot and stick approach.

Conclusion

In my subsequent articles, I’ll try to delve deeper into the practical strategies and steps organisations can adopt and make it more Salesforcy.

Delivery is like a Relay Run As promised in my earlier blogs. I am writing a blog a month and I write about Architecture, Governance and Changing the Process or the Imp

Foute wissel stak stokje voor medaille | NOS
Faulty handover costs gold medal

As promised in my earlier blogs. I am writing a blog a month and I write about Architecture, Governance and Changing the Process or the Implementation and last month was about Technical debt. This month it is about going live successfully. Why? The relay run that the Dutch team lost due to a faulty handover got me thinking about software and delivery, going live, handover moments and risk mitigation. Next to training to sprint really fast, they also train the handover extensively. And despite all this training, this time it went wrong. On an elite Olympic Athlete level!

To be fair, there are many examples of handovers going wrong:

balcony on wrong side of building
And nobody noticed or said anything?

Processes, tools and safety measures

Successful projects have certain elements and key traits in common. These traits consist of

  • Mature, agreed upon processes with KPI’s and a feedback loop to standardize the handovers
  • Automation to support these processes
  • Safety measures for Murphy’s law (when even processes can’t save you)

Key principle is to not try to drown an organisation in red tape and make it more complicated than necessary. Like my first blog “Simplify and then add lightness”. We need these processes to progress in a sustainable and foreseeable way towards a desirable outcome: go live with your Salesforce implementation.

These processes are there to safeguard the handovers. The part of the Dutch relay team that made me think about our own relay runs and their associated risks.

Handovers

The main handovers are:

User → Product Owner → Business Requirement → User Story → Solution Approach → Deploy → User.

As you can see it is a circle and with the right team and tools it can be iterated in very short sprints.

User → Product Owner

“If I had asked people what they wanted, they would have said faster horses.”

Henry Ford

Okay, so there is no evidence that Ford ever said something similar to these words, but they have been attributed so many times that he might as well have said them. I want to use that quote to show different methods on how to get to an understanding of the User Needs. The two sides are either innovating by tightly coupled customer feedback. Or by visionaries who ignores customer input and instead rely on their vision for a better product.

Having no strong opinion on either approach, I still tend be a bit more risk averse and like to have feedback as early as possible. This is perhaps not a handover in a true sense that you can influence as an architect, but getting a true sense of Users Needs might be one that is essential for your Salesforce project to succeed.

I still remember the discussion with a very passionate Product Owner: We need a field named fldzkrglc for storing important data. When diving deeper we found it was a custom field in SAP that was derived from the previous Mainframe implementation. So, that basically meant that the requirements where 50 years old. Innovation?

Business Requirement → User Story

User Stories for Dummies
User Stories for Dummies

There are many ways the software industry has evolved. One of them is around how to write down User Needs. A simple framework I use for validating User Stories are the 3C’s. Short recap:

  • Card essentially is the story is printed with a unique number. There are many tools for supporting that.
  • Conversation is around the story that basically says “AS A … I WANT … SO THAT I …” It’s a starting point for the team to get together and discuss what’s required
  • Confirmation is essentially the acceptance criteria which at a high level are the test criteria confirming that the story is working as expected.

Often used measurement is the Definition of Ready (DoR). It is a working agreement between the team and the Product Owner on what readiness means. And it is a way to indicate an item in the product backlog is ready to work.

As handover and risks go, the quality and quantity of the user stories largely determine the greatness of the Salesforce implementation. Again as an architect you can influence only so many things but in order to bring innovation and move fast User Stories are key.

User Story → Solution Approach

This is where as an architect you can have a solid impact. This is where your high level architecte, solution direction and day to day choices come together. This is your architecture handover moment. When you work together with the developers and create the high level design based on the actual implemented code base. The group as a whole can help find logical flaws, previously wrong decisions and tech debt. The architecture becomes a collaboration. As I wrote earlier, keep it simple and remember Gall’s law. It explains the why of striving for as few number of parts in your architecture.

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.”

John Gall: General systemantics, an essay on how systems work, and especially how they fail, 1975

Next to keeping it simple, I also firmly believe that there should be a place to try out and experiment with the new technology that Salesforce brings. The earlier mentioned experimenting phase fits perfectly. Why only prototype the new business requirements? It is a great place to test out all new cool technical things Salesforce offers like SFDX, Packages or even Einstein and evaluate their value and impact they could have on your Salesforce Org.

Deployment

In any software development project, the riskiest point as perceived by the customer is always go-live time. It’s the first time that new features come into contact with the real production org. Ideally, during a deployment, nobody will be doing anything they haven’t done before. Improvisation should only be required if something completely unexpected happens. The best way to get the necessary experience is to deploy as often as possible.

“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less.”

David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

So establish a repeatable process for going live and perform it many times. This sounds easy but remember that during an Olympic relay race it still went wrong.

Salesforce Sandboxes and our Scratch Orgs provides a Target org to practice your deployments. They are meant for User Acceptance tests, but also making sure that everything will deploy successfully. It can also give developers necessary experience and feedback of deploying their work while it’s in progress. So now that we have a target we need tools to help manage the drudgery.

There are whole suites of tools specifically to support the development team in this. From Gearset to Copado, and Bluecanvas to Flosum. There is a lot, there are even teams that support the business with their own build toolset with Salesforce DX. It is a good practice to choose a tool that supports you and your go-live process to help automate as much as possible.

Safety measures

We have an agreed upon working processes, we measure the quality of handover moments, we automated deployments with bought or homegrown tools, now what?

Even Olympic Athletes make mistakes, so what can we do with Software and Databases that in the physical world is impossible? Backups!

A lot of Salesforce deployments especially for larger customers tend to be fairly data driven. Next to business data as Accounts, Contacts and Orders, there is configured business rule data for example with CPQ. Next to that there is technical data or metadata that is meant for Trigger and Flow frameworks, internationalisation and keeping local diversifications maintainable.

Deploying this data or even making changes in a Production Org is asking for Backups. A good practice is to have a complete Org Backup before you release.

Key takeaways?

  • Establish a process and start ‘training’ your handover moments
  • Automate your release process as much as possible and measure your progress
  • When a handover goes wrong have some safety measures in place

AI and its intrinsic link with data governance I hope you found my previous blog post on AI governance and its link with data governance insightful. The feedback and en

As promised in my earlier blog. I will try to write more often and will write about Changing the Process or the Implementation and Technical debt. Technical debt? What is that? It is a phrase from the famous software developer, Ward Cunningham. Who besides to being one of the authors of the Agile Manifesto, is also credited with inventing the wiki. He used the metaphor to explain to non-technical stakeholders why he needed budget for refactoring existing software.

He didn’t realize at the time, but he had coined a new buzzword in the software community. Later, it would become the subject of many articles, debates and opinions on how to solve technical debt in your code base. In this short blog I want to address the key elements in managing the creation of it in your Salesforce Org. After all, prevention is the best cure.

Types of Technical Debt

So what is it? In my opinion there are many types of technical debt. They range from code that needs to be refactored, application parts that need to be restructured, towards interfaces that work and nobody want to touch, or even complete systems that no longer fit the future application architecture. Who hasn’t seen, in a large company, more than one Enterprise Service Bus that nobody dared to change?

I still remember a factory that had a Dutch team traveling in the US looking on eBay for second hand parts for their server in the Netherlands. Because the company that originally made the parts went bankrupt. That specific server was EOL for more than 10 years. Unmanaged technical debt will become a very serious liability over the years.

Is Technical Debt so bad?

So now that we have a working definition of Technical Debt I want to talk about if it matters. Many of my personal achievements have started with a debt. For example getting my degree meant getting a student loan and buying my first house came with a mortgage. Approached that way, debt is a way to quickly start and defer the full payment. It is notavoiding that payment!

It is hard to predict if starting quickly and incur a debt is a wise decision or that it will be a sound investment. The key point is that it is a debt. Like my colleague said many years ago, it comes down to payment : “Either you pay now, or you pay later, but then? It will be more”.

Debt? To who?

If we agree that having a debt is not that bad as long as you plan on paying. Then who do we pay? Who is paying? The part that makes the problem harder, is the vagueness of the term debt. It immediately makes me think about the creditor and owner of that debt. For the earlier mentioned student loan in the Netherlands, it is clear, I had to pay the government back in a certain time frame. For a mortgage it is also clear, it is the bank. And there are many rules, regulations and stipulations to safeguard those organisations.

But with Salesforce it is often unclear who you have to pay. Is it Future Me? The Enterprise Architecture team or even one of their governance bodies, the Design Authority the creditor? And who is the one that needs to pay? The CIO? The Solution Architect? The Product Owner? The development team? As you can see it can and will vary a lot. As a true consultant I think it depends on the level of the Technical Debt. An Application not fitting in the landscape is probably owned by the EA team. Enterprise Service Bus not being updated in years is a CIO problem and the more small debts should be solved in the teams maintaining or actively developing the applications.

It will boil down to the agreed upon Governance in your organisation: what are the agreed upon rules safeguarding all parties and who gets to decide when to pay to whom?

Is Technical Debt unavoidable?

As stated in the beginning, this blog is about managing the Technical Debt in your Salesforce Org. Let’s be honest, some of these considered Technical Debt issues aren’t created. Often it is the older technology or previously written code that now hinders the requested changes. It still works and does what it needed to do, but is no longer the best fit or optimal solution for the changed requirements.

All software in the end will go the way of the Dodo. It will become obsolete because of many different reasons. Salesforce implementations aren’t that different. There are not that many greenfield implementations, most of us work on existing Orgs that have had many teams with different skill levels working on it. On top of that unstable base there is the phenomenon of decisions made in the past, being overtaken with the technology push of today. When I joined Salesforce we still had the Classic UI. And now 5 years later there are still customers that haven’t transitioned yet.

What are key drivers of Technical Debt in your Org?

I believe these are the main aspects that contribute

  1. Business pace
  2. Agile everything
  3. Tech lead Technical debt

Let’s explore each topic:

Keeping up with Business

Business will always want to go at a pace IT can’t keep up with and in my opinion that’s the way it should be. That is what we try to solve with Pace Layering our Enterprise Architecture. Not all elements of an application landscape need to be changed with every season. For example: websites will need to change more often than sending invoices to the customer. Stuart Brand did some great explanation of applying Shearing Layers inside applications. The question that I think lies at the heart of it is: Should every request or new idea be implemented immediately? That is a great way to accrue technical debt on a more business architectural level.

Solution
In general it is a good practice to explore and test innovations and new ideas for the value that they promise before determining if they can and should scale up and end up as projects and items on the backlog. In some organisations there is a role that acts as the Portfolio lead that can safeguard how many projects will be invested in and can be executed at the same time while still delivering the expected value. Managing the different priorities is an on-going aspect and not a one-time resolution. It’s just part of the process. Next to this more Governance approach I often see a Technical Architect that with a small team develops these ideas in so called Proof of Concepts or Architectural Spikes on a separate Sandbox to validate expectations, solution approach, technical fit with existing Org and overall value.

Organised this way you will have a safe place for experimenting and trying to see if you can make it work in your Salesforce Org. No is still a valid answer!

Agile solves everything, right?

Agile is a way to take bite sized chunks of the agreed upon strategy to implement the long term vision for a specific product, not a replacement for strategy. It lets you pivot quickly, so you can recover from a bad decision or implement features which suddenly get a higher priority. Next to that, some architectural decisions are too big and important to leave until the last moment. If different Agile teams do not coordinate or collaborate towards that shared architectural understanding of where they fit in and how to achieve outlined business goals that is another way to accrue technical debt.

Solution
Agile can be seen as a Risk Management approach by staying in close communication with all your key stakeholders and show short term progress towards the larger picture on your key deliverables. You need to strike a balance between just enough architecture and upfront planning for critical, large and complex projects.

The earlier mentioned ongoing tinkering, prototyping, refactoring and architecture experiments are an important part. They can, for example, validate the Architecture decision that a new real time integration pattern will be solved by using Platform Events.

Cross team alignment for Agile teams is also needed to prevent Technical Debt. Different teams need to stay aligned on the architecture, the business strategy and how to manage those deliverables across teams. There are many solutions. From big room planning, or shared backlogs to even complete methodologies like SAFE. They all should lead to an agreed upon and prioritized roadmap that is then input for the different teams, their Product Owners and their backlogs. Managed in this way the big architecture topics don’t popup out of nowhere. They are managed from the start across the teams.

Tech lead Technical Debt

Sometimes I encounter a sort of disconnect in development teams and the business partner. Not physical but more the sentiment that “the Business” is not clearly stating their requirements. Or that they haven’t thought it through, so instead of sticking to a consistent implementation. And iterating and refactoring the current design we have a Tech lead gold-plating solutions to requirements. We just need an interface here, trust me in two or three months we will definitely need the extra abstraction it provides. I saw this really cool injection pattern I want to try out. And over there we need event driven updates, so we are totally ready for when users will actually use it…

Let’s be honest, most people tend to be less interested in the needed complex technical elements that make a feature work. And thus relative simple features may add disproportionately to a development schedule. And that is hard to explain, everytime. So it makes sense to want to be prepared, but at the same time you have to wonder how far are you willing to go and for what?

Next to that, most Architects and Developers that I know are fascinated by new technology and are itching to try out the new features of Salesforce and create their own implementation of a slick new thing they saw. Whether or not it’s required.

Salesforce releasing three times a year adds to that pressure of both wanting to implement the new innovative technology, but also deprecating some older implementations. For example: Process Builder and Workflows being migrated to Flow is a great example of the last category.

Solution
The solution to this technology driven Technical Debt is two fold.
I firmly believe that there should be a place to try out and experiment with the new technology that Salesforce brings. The earlier mentioned experimenting phase fits perfectly. Why only prototype the new business requirements? It is a great place to test out all new cool technical things Salesforce offers like SFDX, Packages or even Einstein and evaluate their value and impact they could have on your Salesforce Org.

Next to that there is a need for an ongoing process of evaluating and adjusting your current implementation standards and Org Maintenance. From newly needed naming conventions for Flow to getting a standard for Error Handling and Logging Errors.
The second part is that we all need to communicate better. Sometimes infrastructural elements need to be in place before something fancy can be build. If your Salesforce Org is not yet able to support the requested feature what is the plan to get there?

This all leads to different levels of elements in the roadmap and the backlogs. Both Business User Stories and underlying supporting Technical User Stories in the backlog. Without the proper planning and attention for both, Technical Debt will accrue with every Salesforce release even if you have the best Salesforce Admin in the world.

Key takeaway

Is Tech Debt avoidable? It will be better manageable if you pay attention to

  • Proper maintenance of your Org as the standards
  • Keeping pace with Salesforce releases
  • Balance the push for new features with proving their value
  • Experiment with improvements in your PoC environment

“Simplify, then add lightness” This famous quote is from Colin Chapman. It was his philosophy, somewhere in 1950 way before ‘minimalism’ became fashionable. Least numbe

This famous quote is from Colin Chapman. It was his philosophy, somewhere in 1950 way before ‘minimalism’ became fashionable.

Colin Chapman – engineer, inventor, and builder in the automotive industry, and founder of Lotus cars

Least number of parts

By tradition, Lotus uses the least number of parts needed in its products. Yet, they are impeccably engineered, retain their lightness and work dependably.

This is a great analogy and one we need to see more often in our approach to IT systems. It is what we should strive to establish with our architecture and designs. Yet we often see the opposite. We add many abstraction layers of indirection to be ready for possible future changes.

But when we discuss requirements or features we use the term MVP. Why not with our designs and architectures? Design is driven from requirements and should fit the overall architecture. Building anything beyond those requirements is pure speculation. My goal is to make the statement of Colin Chapman “Simplify, then add lightness” the guiding principle when architecting for a project.

Thinking about how to establish a wider practice around the concept of keeping it simple, lean and minimal, Gall’s law also comes immediately to my mind. It explains the why of striving for as few number of parts in your architecture.

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.”

John Gall: General systemantics, an essay on how systems work, and especially how they fail, 1975

Although dubbed Gall’s Law by some, the phrase is not labeled as such in the original work. The book cites Murphy’s Law and the Peter Principle, and is filled with similar sayings. Gall’s Law is a rule of thumb for systems design from Gall’s book Systemantics: How Systems Really Work and How They Fail.

As a Salesforce Program Architect I’m involved at some of our biggest customers and their large and complex implementations. I’ve heard lots of stories about failed Salesforce implementations and they all had something in common. It became an over-built system designed for the wrong people and processes.

I can relate. It is a constant struggle to keep it as simple as can be and still deliver fast. Part of the struggle is our own knowledge. We know that some design patterns lead to brittle, hard to change systems with lots of Technical Debt. That will be part of another blog. Other part may be the question if the proposed first release is appealing enough? Yet another part may be the impact of keeping it simple. Do you change the design or change the process in order to keep it simple and deliver fast? There is certain inertia that comes with change. But that also will be part of another blog post.

Coping strategies

Why do I like to use Chapman and Gall? Chapman has great quotes that are understandable and translate well to the project. “In order to increase speed, you have to add lightness”. This sparks great discussions with the customer. How can we make this simpler and still have value? What is the smallest increment that we can deliver and is deemed valuable?

I like Gall’s law because influential people I learned a lot from in my career mentioned Gall’s law. One of the first times I encountered Gall’s Law was when Grady Booch mentioned it. I don’t need to explain Grady, right? Right?

What is also great is that in later books Gall even states some very familiar strategies to scale up from the simple system and cope with the possible negative outcomes. Sound familiar?

Develop using Iterative processes

Building iteratively is about reaffirming or refining the shared understanding of the problem. And getting feedback fast from actual users. You can think of it as a trial-and-error methodology that brings your project closer to its end goal.
Build a MVP that addresses the most important issues first. Save the extra stuff for later. A small release can help get your users involved more quickly and generate better feedback without the risk of over-architecting, over-designing and overbuilding.

Reuse known working modules

Avoid over-doing by using standard Salesforce features. Salesforce consists of many known working modules and those connect really well. For example: Lead management, Account Management to Opportunity and their scoring model and Case Management. There is no need to architect, design or develop this all by yourself. These modules still need to be configured and sometimes even extended by some customization. This needs to be done by well known ‘good practices’. But that probably also should be YABP (yet another blog post). Salesforce has great resources on well known patterns and practices, visit https://architect.salesforce.com/ to learn more.

Release early and often

In any software development project, the riskiest point is always deployment time. It’s the first time that new features come into contact with the real production org. Ideally, during a deployment, nobody will be doing anything they haven’t done before. Improvisation should only be required if something completely unexpected happens. The best way to get the necessary experience is to deploy as often as possible.

“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less.”

David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Salesforce Sandboxes and our Scratch Orgs provides the means to do this. They are perfect as target practice to make sure that everything will deploy successfully. It can also give developers necessary experience and feedback of deploying their work while it’s in progress. There are whole suites of tools specifically to address this. From Gearset to Copado, and Bluecanvas or Flosum.

Automate testing

Use automated testing to ensure that enhancing the system does not break it. Otherwise any further steps are enhancing of a non-working system.

“So, when should you think about automating a process? The simplest answer is, “When you have to do it a second time.” The third time you do something, it should be done using an automated process.”

Jez Humble, Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

Building, executing, and maintaining tests can take a long time. How many business logic decision does your Salesforce implementation have? What about the different system and application integrations? To add to the complexity, Salesforce releases many new features delivered by three major upgrades every year. Great for improving functionality, but these regular upgrades break manual scripts, which probably require many hours to fix. All this leads to diminishing returns on manually creating and maintaining your test.

So part of the solution is to automate your test to ensure that Salesforce still delivers what you agreed upon to deliver.
And like the release early and often it also has several companies that deliver solutions that can help you. Depending on your team size and skill levels and of course pricing.

Key takeaway

In order to be successful with your Salesforce implementation: “Keep it simple and add lightness”

  • Start small and simple with your features and design
  • Add more features and functionality based on actual User Feedback
  • Use iterative processes
  • Reuse known working modules
  • Release early, release often
  • Use automated testing

Software as an Onion (SaaO)

http://www.pngpix.com/wp-content/uploads/2016/03/Red-Sliced-Onion-PNG-image.pngSoftware is like an onion. Your team’s code is one “layer” in this onion. There are layers below you (which you rely on), and layers above you (which rely on you).

Of course, unlike an onion in real life, all the layers in this onion are continuously changing. New features, bug fixes, performance optimizations, etc.

In a good architecture, the effect of change flows in exactly one direction: up. You can only break layers above you, not below. This implies that the deeper you are in the onion, the greater the responsibility. Any little change can ripple upward in unexpected ways, because there’s a deep intertwining–a dependency.

https://lifeofgaius.files.wordpress.com/2015/03/ripples_pools2_680x347.jpg?w=869&h=444

It’s also like the princess and the pea; even the tiniest change can make stuff higher in the stack uncomfortable; you never know what behaviors / bugs / etc. the upper layers are relying on.

You can try to avoid breaking stuff at compile time (e.g. changing an API in an incompatible way). Beyond that, you can also run all the unit tests, get code reviews from peers and/orrelevant dependent teams, etc. Isn’t that good enough?

Alas, no. The nature of human fallibility is, things aren’t perfect, and if you plan for them to be perfect, you’ve made an imperfect plan. There will always be bugs, and bugs in lower layers can reach WAY up the stack to hose features and team way above them. This is true of everything even hardware: remember the Intel Meltdown and Spectre bugs?

The only “bullet proof” world is one where teams and services are totally independent.

Unfortunately, this isn’t possible when you have to share. Which we all do of lots of stuff. Housing and hosting, network, VMs, Containers, DB schema’s, Development stacks or even Amazon solutions.

Masterclass Serie 1 “CIO Office anno 2020” – (BE)DENKEN – Van visie naar praktisch uitvoerbaar Actieplan De bijdrage van informatie en IT aan het succes van de onderne

Masterclass Serie 1 “CIO Office anno 2020” – (BE)DENKEN – Van visie naar praktisch uitvoerbaar Actieplan

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

September 23, 2014

De bijdrage van informatie en IT aan het succes van de onderneming neemt alleen maar toe. Reden genoeg om de organisatie die daar over gaat op de juiste manier in te richten en, als dat nodig is, drastisch te veranderen. Deze Masterclasses geven aan welke ‘spelers’ belangrijk gaan worden, welke vraagstukken moeten worden opgelost en in welke richting we de antwoorden moeten zoeken op de vraagstukken die zich aandienen in de (nabije) toekomst. Waarbij samenwerking tussen de disciplines Informatiemanagement, Architectuur en Information Risk Management onont-koombaar is en de beslissers in de IV kolom meer dan ooit het stuur van de onderneming in handen hebben om de juiste koers te varen. De wereld verandert snel en ingrijpend. Een aantal ontwikkelingen die hierbij een rol spelen zijn:
Technology push;
Information push: big data, business intelligence, door data gedreven, personificerende processen, internet of things;
Society push of de crowd push: social media, de crowd die de expert vervangt.
Allemaal veranderingen die sterk zijn verbonden met informatie en IT waarbij de juiste inzet hiervan bepalend is voor het succes van een organisatie.Opzet van de Masterclass SerieMaar hoe ga je hier nu mee om? In een drietal Masterclasses geeft Quint op basis van eenvoudige richtlijnen en concrete praktijk cases van verschillende bedrijven aan hoe ondernemingen hun business- en ICT/IV-strategie (kunnen) formuleren en omzetten in succesvolle realisatie. Reserveer daarom alvast elke laatste donderdag van de maanden oktober, november en december in uw agenda.Donderdag 30 oktober: (BE)DENKEN – Van visie naar praktisch uitvoerbaar ActieplanOrganisatie, informatie en IT raken meer en meer verweven. De eerste Masterclass gaat in op de vraag hoe we een handzame en praktische visie, strategie en richting kunnen definiëren en hoe we die omzetten tot een realistisch en haalbaar actieplan.Voor wie is deze Masterclass interessant?
CIO’s en IT Management
Informatie- en Domeinmanagers
Risk en Security Managers
Architecten
Business Managers betrokken bij Innovatie
Programma & DeelnameWe beginnen om 14:00 uur en sluiten af rond 17:00 uur. Tijdens de sessie zullen er twee presentaties worden gegeven over het onderwerp. Naar aanleiding van die presentaties wordt op interactieve wijze een dialoog gevoerd met elkaar.Deelname is kosteloos. Indien u zich wilt inschrijven voor meerdere Masterclasses uit deze serie, gebruik dan het opmerkingenveld van het aanmeldformulier om uw wensen door te geven. Neem gerust een collega mee!Overige Masterclasses uit deze serie:Donderdag 27 november: DURVEN – Strategie moet je durven uitvoeren Elke organisatie heeft wel een strategie of een missie. Maar wie is er binnen de organisatie concreet bezig met het realiseren van die strategie? En hoe weten we dat we met de juiste dingen bezig zijn en blijven? Het concretiseren van de strategie dwingt de verschillende disciplines (informatie management, architectuur, IRM) in de organisatie om samen te werken. De tweede Masterclass laat zien hoe de rollen en activiteiten van deze disciplines samenwerken om de strategie daadwerkelijk te realiseren. Donderdag 18 december: DOEN – Wat heeft de bedrijfsvoering nodig en waar moet ICT aan (gaan) voldoen? Nieuwe technologie, samenwerken in ketens en een toenemende mix van intern en extern betrokken diensten vragen om een organisatie die niet alleen procesmatig, maar ook inhoudelijk de regie kan nemen. Activiteiten zoals informatiemanagement, information risk management en architectuur zullen zich sterk moeten ontwikkelen. Naast een inkijkje in de toekomstige ontwikkeling van deze disciplines kijkt deze Masterclass juist naar de stappen die u morgen al kunt zetten om in de toekomst het stuur in handen te blijven houden. Deze nieuwe reeks Masterclasses staat in het teken van trends, ontwikkelingen en uitdagingen in ons vakgebied en is gebaseerd op de visie van Quint. Ervaren consultants, betrokken managers en gedreven vakmensen gaan met elkaar in gesprek en in debat om visies met elkaar te delen en ervaringen uit te wisselen.schrijf je nu in

Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices

The Amber Trap

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

August 29, 2025

Last week I was cleaning out some old project folders (a therapeutic exercise I recommend to any one). I learned that from a fellow architect. Does this spark joy? Anywho, I stumbled upon a perfectly preserved architecture document from 2018. Complete with detailed diagrams, stakeholder matrices, and decision rationales that read like artifacts from a lost civilization. The document was comprehensive! And absolutely useless for understanding that current system. It had become digital amber. Beautiful preserved, but tragically misleading.

This got me thinking about my previous article on the contextual decay in software systems. While that article focused on how LLMs miss the living context around code, there’s another phenomenon at play. Sometime we do preserve context, but it fossilizes into something that actively misleads rather than informs.

The Dominican Republic of Documentation

In the mountains of the Dominican Republic, amber deposits contain some of the most perfectly preserved specimens in the natural world. Insects trapped in tree resin millions of years ago look as fresh as if they died yesterday. Their delicate wing structures, compound eyes, and even cellular details remain intact. It’s breathtaking. It’s also a snapshot of a world that no longer exists. And people pay serious money for these risk fraught mined specimens, that is a comple other rabbit hole. Yes I ended up on catawiki.

The forest that produced that resin is gone. The ecosystem those insects lived in has vanished. TThe amber preserves the form perfectly while the function has become irrelevant.

The Amber Trap: When Context Gets Fossilized

This leads me to a fourth law, a corollary to Contextual Decay. I call it

The Amber Trap: A system’s context, when perfectly preserved without evolving, becomes a liability that actively harms its future development.

In my previous article, I argued that code repositories are like dinosaur skeletons. All structure but miss the context. Documentation was supposed to be that soft tissue, filling in the gaps with the why behind the what. But here’s what I’ve realized: soft tissue doesn’t just decay it sometimes it mummifies.

Real soft tissue decomposes quickly, which is actually helpful. It forces paleontologists to constantly seek new evidence, to challenge their assumptions, to remain humble about their reconstructions. See https://www.nhm.ac.uk/discover/news/2025/august/bizarre-armoured-dinosaur-spicomellus-afer-rewrites-ankylosaur-evolution.html

Outdated Architecture Decisions

I’ve seen this pattern repeatedly across organizations:

The Authentication Fossil: A security architecture document from 2019 explaining why we chose a custom JWT implementation instead of OAuth because “we need fine-grained control and the OAuth landscape is too fragmented.” Today, that same custom implementation is our biggest security vulnerability, but new developers read the document and assume there’s still a good reason for not using industry standards.

The Database Decision Artifact: Documentation explaining why we chose MongoDB because “we need to move fast and can’t be constrained by rigid schemas.” The current system now has more validation rules and data consistency requirements than most SQL databases, but the document makes it seem like any proposal to add structure is fighting against architectural principles.

Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices

Yes, the zeroth law!

Engineering doesn’t solve problems…

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

July 10, 2025

We just trade them in for newer, more interesting ones. And that’s our real job.

You ever spend a week with your team fixing a critical system, finally push the fix live and feel that wave of relief? Only to get a Slack message two months later. “Hey, weird question. Ever since we deployed that one patch, the reporting dashboard runs… backwards?”

Of course it does.

This is a beautiful lie

We have a sacred belief that we are problem solvers. We take a messy world and make it clean with elegant logic. We’re not problem solvers. We are professional Problem Shifters. Think about it. We “fixed” monolithic backends… and created the 1,000-microservice headache that is only stable in the weekends as no one is deploying (Real story from Uber Lead Architect). We “fixed” manual server deployments… and created the 3000-line YAML file. Therefore we now need anchors and aliases.

It makes me think of the invention of the refrigerator. It’s a modern miracle. On the other hand, the cooling liquid tore a hole in the ozone layer.

That’s our job in a nutshell. We aren’t creating a utopia of solved problems. We’re just swapping today’s problem for tomorrow’s way more fascinating crisis. Yeah, I’m old and I was there to fix the year 2000 problem.

My PTO is coming up, and I was looking at books to take with me. Next to Bill Bryson’s a short history of nearly everything I also packed Hitchhikers guide to the galaxy by Douglas Adams. I’ve read them before, but never combined them.

I think it is safe to say that we have found the :

First Law of Engineering Thermodynamics

Problem energy can neither be created nor destroyed, only changed in form.

Saying this out loud is a bit cynical. But I think you can also make it into a superpower. If we are smart about our entire approach.

We need to Stop chasing “done.” And start chasing “durable.” The goal isn’t just to close the ticket. It’s to create a solution that won’t spawn five more tickets.Ask the “Next Question.” The most important question is “if we build this, what new problem are we creating?” Acknowledge the trade off upfront.Redefine your win. The best engineers aren’t the ones who solve the most problems. They’re the ones who create the highest quality of future problems. That did not come out right. It’s having less complexity and less problems.

Building stuff is easy. The same goes for adding quick fixes, work arounds or low hanging fruits. Building a maintainable future that’s the hard part.

What’s the best “future problem” you’ve ever created? Drop your best story in the comments.

Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices

Beyond the Policy pdf

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

June 27, 2025

An Architect’s Guide to Governing Agentic AI

Your company is rushing to deploy autonomous AI. A traditional governance playbook won’t work. The only real solution is to architect for control from day one.

You can feel the pressure on CTOs and Chief Architects from every direction. The board wants to know your AI strategy. Your development teams are experimenting with autonomous agents that can either write their own code. Or access internal systems, and “oh no!” interact with customers. I understand. The marketing engine is working overtime and the promise is enormous: radical efficiency, hyper-personalized services, and a significant competitive edge.

But as a CIO or CTO, you’re the one who has to manage the fallout. You’re the one left with the inevitable governance nightmare when an autonomous entity makes a critical mistake.

Statement of intent

Let’s be clear: the old governance playbook is obsolete. A 20-page PDF outlining “responsible AI principles” is a statement of intent, not a control mechanism. In the age of agents that can act independently, governance cannot be an afterthought! I strongly believe it must be a core pillar of your Enterprise Architecture.

This isn’t about blocking innovation. It’s about building the necessary guardrails to accelerate, safely.

The New Risk: Why Agentic AI Isn’t Just Another Tool

We must stop thinking of Agentic AI as just another piece of software. Traditional applications are deterministic; they follow pre-programmed rules. Agentic AI is different. It’s a new, probabilistic class of digital actor.

Great example I heard

AI interpretation of Agents running around like Lemmings

Think of it this way. You just hired a million-dollar team of hyper-efficient, infinitely scalable junior employees. But they have no (learned) inherent common sense, no manager, and no intuitive understanding of your company’s risk appetite. Makes me think of the game Lemmings?

This looks a lot when IaaS, PaaS & SaaS and moving towards the Cloud was being discussed, but with something extra:

Data Exfiltration & Leakage: An agent tasked with “summarizing sales data” could inadvertently access and include sensitive data in its output, as seen when Samsung employees leaked source code via ChatGPT prompts.Runaway Processes & Costs: An agent caught in a loop or pursuing a flawed goal can consume enormous computational resources in minutes, long before a human can intervene. The $440 million loss at Knight Capital from a faulty algorithm is a stark reminder of how quickly automated systems can cause financial damage.Operational Catastrophe: An agent given control over logistics could misinterpret a goal and reroute an entire supply chain based on flawed reasoning, causing chaos that takes weeks to untangle.Accountability Black Holes: When an agent makes a decision, who is responsible? The developer? The data provider? The business unit that deployed it? Without a clear audit trail of the agent’s “reasoning,” assigning accountability becomes impossible.

A policy document can’t force it to align with your business intent. The only answer is to build the controls directly into the environment where the agents live and operate.

Architecting for Control

Instead of trying to police every individual agent, a pragmatic leader architects the system that governs all of them.

Pillar 1: The Governance Gateway

Before any agent can go live executing a significant actions like accessing a database, calling an external API, spending money, or communicating with a customer. It must pass through a central checkpoint. This Governance Gateway is is where you enforce the hard rules:

Cost Control: Set strict budget limits. “This agent cannot exceed $50 in compute costs for this task.”Risk Thresholds: Define the agent’s blast radius. “This agent can read from the Account object, but can only write to a Notes field.”Tool Vetting: Maintain an up to date “allowed list” of approved tools and APIs the agent is permitted to use.Human-in-the-Loop Triggers: For high-stakes decisions, the design should automatically pause the action and requires human approval before proceeding.

This is should be a familiar concept familiar because I borrowed it from API gateways, now just applied to Agentic actions. An approved Design is your primary lever of control.

Pillar 2: Decision Traceability

When something goes wrong, “what did the agent do?” is the wrong question. The right question is, “why did the agent do it?” Standard logs are insufficient. You need a system dedicated to providing deep observability into an agent’s reasoning.

This system must capture:

The Initial Prompt/Goal: What was the agent originally asked to do?The Chain of Thought: What was the agent’s step-by-step plan? Which sub-tasks did it create?The Data Accessed: What specific information did it use to inform its decision?The Tools Used: Which APIs did it call and with what parameters?The Final Output: The action it ultimately took.

This level of traceability is non-negotiable for forensic analysis, debugging, and, crucially, for demonstrating regulatory compliance. It’s the difference between a mysterious failure and an explainable, correctable incident.

Pillar 3: Architected Containment

You wouldn’t let a new employee roam the entire corporate network on day one. Don’t let an AI agent do it either. Agents must operate within carefully architected contained environments.

This goes beyond standard network permissions. Architected Containment means:

Scoped Data Access: The agent only has credentials to access the minimum viable dataset required for its task.Simulation & Testing: Before deploying an agent that can impact real-world systems, it must first prove its safety and efficacy in a high-fidelity simulation of that environment.

Containment isn’t about limiting the agent’s potential; it’s about defining a safe arena where it can perform without creating unacceptable enterprise-wide risk.

From Risk Mitigation to Strategic Advantage

Building this architectural foundation may seem like a defensive move, but it is fundamentally an offensive strategy. This first version of the framework is more than a set of features; it’s a strategic shift. It allows you to move away from the impossible task of policing individual agents and towards the pragmatic, scalable model of architecting their environment.

This is how you build a platform for innovation based on trust, safety, and control. It’s how you empower your organization to deploy more powerful AI, faster, because the guardrails are built-in, not bolted on.

The organizations that master AI governance will be the ones that can deploy more powerful agents, more quickly, and with greater confidence than their competitors. They will unlock new levels of automation and innovation because they have built a system based on trust and control.

This architecture transforms the role of EA and IT leadership. You are no longer just a support function trying to keep up; you become the strategic enabler of the company’s AI-powered future. You provide the business with a platform for safe, scalable experimentation.

The conversation needs to shift today. Stop asking “what could this AI do?” and start architecting the answer to “what should this AI be allowed to do?”

What’s your take? Of these three pillars: Gateway, Decision traceability, and Containment: which presents the biggest architectural challenge for your organization right now?

Share your thoughts in the comments below.

Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices

It’s Time to Play the Music

Martijn Veldkamp

“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”

April 17, 2025

Ever felt your design review was less constructive critique and more… relentless heckling from a balcony?

A couple of years ago, when I worked at Quint I wrote an article together with Rob Swinkels for the magazine called Informatie. It was based on the ideas of Mark de Bruin who stated that as an architect you should be aware of the role you choose in your assignment. We took that a step further (with his permission) and called it from Einstein to Mandela and added a bit of Circle of Influence and other roles. Even Livingstone!

While still a great article, perhaps the day-to-day reality of architectural life finds a more fitting comparison in the cast of The Muppet Show. Its beautiful chaos, clashing personalities, and its moments of sheer, unadulterated absurdity! Chocolate Moose…

The architectural profession, often perceived as somewhat ‘ivory towerish’, is certainly multi-faceted. While discussing spirit animals with my kids, my inner curmudgeon came out and said Waldorf and Statler. This got me thinking.

This article is a deliberately nonsensical, tongue-in-cheek look in the mirror, using the Muppet Show finest as a warped lens. Consider this a unnecessary continuation of that earlier piece and thus a bit of “Friday nonsense”. It’s a personal antidote to the serious side of business. Don’t you have meetings that you wish you just could end with a karate chop?

I had this article in draft for quite a while and was having way too much fun describing the characters and rewatching episodes.

Kermit the Frog

The Ever-Patient (and Slightly Panicked) archetype

Leading, or perhaps just trying desperately to keep the curtain from falling, is Kermit. As the de facto leader of his chaotic troupe, he’s more the group’s glue than an authoritarian figure. He embodies the Project Manager: kindhearted, polite even when faced with absurdity. Constantly trying to find balance among the clash of personalities and their demands. He leads through dreams and a desire to help his friends succeed. Can he keep pace with the backstage commotion? The high-chaos environment of architectural practice on all levels? He mediates conflicts between demanding Starchitects (Piggy-types), unpredictable creatives (Gonzo/Animal-types), and nit-picking reviewers (Statler & Waldorf-types).

He holds it all together, even if the binding agent is mostly sweat and existential dread. Okay, I’m making that up. Under extreme pressure, though, even Kermit can become cynical or bossy, revealing the strain beneath the calm green exterior.

Miss Piggy

The ‘Moi’-numental Starchitect type.

Making her grand entrance is Miss Piggy, the embodiment of the ‘Starchitect’ and a great example of the Prima Donna stereotype. She is convinced of her destiny for stardom and possesses a temperamental diva personality. Criticism is not tolerated. And any perceived slight might be met with a swift “Hi-yah!” karate chop.

French phrases are adding an air of sophistication to statements like, “Kermie, darling, the client’s budget is merely a suggestion, non? The Starchitect’s aggressive defense of their vision could potentially mask a deeper need for validation. In a profession where egos can be both large and fragile, sometimes appearing simultaneously, the diva-like behavior might serve as a performative shield against insecurity.

Fozzie Bear

The Relentlessly Optimistic

The insecure stand-up comic and relentlessly enthusiastic concept pitcher. He desires nothing more than to make people laugh. And be loved by his audience. He presents ideas with infectious enthusiasm often punctuated by his hopeful catchphrase.

His jokes are often met with groans, particularly from Waldorf and Statler and. Fozzie’s takes criticism hard and feels crushed by negative feedback. He looks to his best friend Kermit for leadership and validation, his approval making or breaking Fozzie’s day. Fozzie embodies this emotional risk, making supportive team dynamics crucial to weather the inevitable ‘heckling’.

Gonzo the Great

The Visionary

Then there’s Gonzo, the performer of dangerous stunts. As an archetype, he’s the experimental designer. He embraces weirdness, seeking artistic meaning in chaos. His plans might resemble performance art manifestos more than design decision documents.

Yet, Gonzo isn’t just weird for weirdness sake. Beneath the zany stunts lies a surprising sensitive, philosophical, and a melancholic persona;ity. He has an almost poetic desire to create something meaningful and unique.

Animal

Instinctual designer

Animal operates on raw energy and instinct. He’s the chaotic creative. And has trouble communicating design intent. “DESIGN! GOOD! BUILD! NOW!”.

He often needs supervision (like his bandmate Floyd keeping him chained to the drum kit). The Animal architect ignores constraints, budgets, and sometimes physics. Yet, Animal is also a savant, a versatile and capable musician when focused. Occasionally, the Animal architect produces a flash of brilliance, usually by accident, amidst the chaos. The key is finding the right context and guidance to channel the raw energy away from pure destruction towards productive, if still LOUD, design.

Statler & Waldorf

The Balcony Critics

AKA the jaded senior architect who’s seen it all and is thoroughly unimpressed. They find fault with everything, often with sarcastic glee, bursting into laughter at their own witty criticisms.

They embody the resistance to change, the mentally retired attitude sometimes encountered in long-established figures. They’ll question every decision and predict project failure with smug satisfaction. Despite constantly complaining about the show, they always return for the next performance. Their consistent attendance despite their negativity suggests their criticism might be somewhat performative, stemming from their own fixed perspectives and enjoyment of heckling, rather than evaluation aimed at improvement.

This archetype represents criticism potentially rooted in defending the status quo or asserting authority, rather than fostering innovation.

Swedish Chef

The Bork-tastic

Known for his incomprehensible mock-Swedish gibberish. And crazy cooking methods involving bizarre tools like firearms, he represents to me the highly specialized technical architect. His explanations are impenetrable technical jargon “Urdy Boordy, flipperdy schnurde, de R-value!”.

Communication inevitably breaks down, often ending with something exploding (“Bork, bork, bork!”). The Chef’s combination of incomprehensible jargon and unconventional methods humorously reflects the communication gap that can arise between technical specialists and the broader team.

While potentially innovative, their expertise can become counterproductive if not translatable or practical.

No actual Muppets (or architects) were harmed in the making of this article

Perhaps there’s a grain of truth in the madness. Do you recognize the frazzled but determined Kermit in your project, trying to steer the ship through stormy seas? Have you encountered the diva demands of a Miss Piggy-esque designer? Or felt the crushing weight of Statler and Waldorf’s critiques during a presentation?

Maybe, just maybe, you see a bit of Fozzie’s desperate need for approval in one of your own pitches, or Animal’s raw energy in a caffeine-fueled deadline sprint? Or perhaps you’re the Gonzo, passionately advocating for a truly bizarre idea, wondering why no one else quite gets your vision for AI agents deploying self build LWC components?

Ultimately, the comparison is ridiculous. Yet, sometimes, embracing the ridiculous is the only way to maintain sanity when the deadline looms large, the client is heckling from the virtual balcony, and someone (Animal!) appears to have broken every unit test in the system.