Herding Cats, Seeding Clouds
Yesterdays post by Bard Papegaaij talked about the often used statement “Culture eats Change for breakfast”. And that got me thinking about
AWS’s S3 outage
AWS’s S3 outage
Martijn Veldkamp
“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”
March 1, 2017
Tuesday’s Amazon Web Services mega-outage affected not only websites big and small, by disrupting their backend storage, but also a lot of apps and Internet of Things gadgets relying on the technology. The AWS storage offering provides hosting for images for a lot of sites, and also hosts entire websites, and app backends including Nest.
In fact, the five-hour breakdown was so bad, Amazon couldn’t even update its own AWS status dashboard: its red warning icons were stranded, hosted on the broken-down side of the cloud.
The S3 buckets in the US-East-1 region became inaccessible at about 0945 PST (1745 UTC) taking out a sizable chunk of the internet as we know it.
AWS has many many regions, and US-East-1 is just one of them. Developers are supposed to follow Disaster Recovery architecture and Best Practices and spread their applications over different data centers.
For various reasons – from the fact that programmers find distributed computing hard to the costs involved – this redundancy isn’t always coded in. And so here we are.
Stolen from https://static.vecteezy.com/
Stolen from https://static.vecteezy.com/
From Blueprint to CTO
Martijn Veldkamp
“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”
February 14, 2025
Remember those old cartoons where the architect with a rolled-up blueprint and a pencil behind their ear, stood stoically overseeing construction?
That image, while nostalgic, is as outdated as dial-up internet. The role of the architect has undergone a dramatic transformation, evolving from a technical specialist to a vital business strategist.
Beyond Blueprints
In todays fast-paced digital world, architects are no longer just drawing up technical plans. They’re navigating a complex application landscape of cloud computing, artificial intelligence, data analytics, and ever-shifting business needs.
The blueprint is still important, but its now just one piece of a much larger puzzle.
Think of it like this: Imagine building a house. Architects used to focus primarily on the structural integrity, ensuring the walls wouldn’t fall down. Now, you also need to consider energy efficiency, smart home integration, and how the house will adapt to the family’s changing needs over time. The modern architect is concerned with the entire ecosystem over time, not just the foundation.
Captain Sparrow fan art
Business-Savvy! As an architect you need to understand the business inside and out. How to translate business goals into technical solutions, and vice versa. You need to be fluent in the language of both the C-suite and the development team.
This means understanding market trends, competitive pressures, and how technology can drive innovation and create a competitive edge.
Its no longer enough to simply design elegant systems. The architect must also consider the business impact of their decisions. Will this architecture enable faster time to market? Will it reduce costs? Will it improve customer experience? These are the questions that keep CEOs awake at night, and architects need to have the answers.
The Architect as a Change Agent
And not a weapon of mass confusion
You are also playing a crucial role in driving digital transformation. Helping businesses adopt new Salesforce technologies, migrate away from their legacy systems, while embracing agile methodologies. This requires not only technical expertise, but also strong leadership, communication, and change management skills.
The architect is no longer just a technical expert; they are a change agent, helping organizations navigate the complexities of the digital age.
The Customer CTO
The days of the lone wolf architect working in isolation are long gone. Todays architect is a collaborator, working closely with developers, business analysts, security experts, and other stakeholders. You need to be able to build consensus, negotiate compromises, and foster a culture of collaboration.
The modern architect understands that the best solutions are often the result of diverse perspectives and shared knowledge.
That is why I think that CTO can mean much more then just Chief Technology Officer:
Collaborative Transformation Officer, Consulting Technology Orchestrator, Connector of Technology & Organizations, Catalyst for Technology & Operations
I wanted to be even more poetic but that is quite hard with just these letters. I also kinda like the following, but I’m ending the Friday with a beer and a blog, so bear with me:
Weaver of Technology & BusinessCurator of Technological FuturesInnovator of Technological Approaches
To state some open doors
The world of technology is constantly evolving. New technologies emerge, old technologies become obsolete, and best practices are continuously being redefined. You as an architect must be a continuous learner, always seeking to expand their knowledge and stay ahead of the curve.
This means attending conferences, reading industry publications, experimenting with new technologies, and engaging with the broader architectural community. Personally I mostly have several authors that I really like that help shape my thoughts. Martin Fowler has a nice group of people that use his platform to distribute interesting ideas.
To try and close the article. Organisations will need people that can act hybrid, blending technical expertise with business acumen, leadership skills, and a passion for innovation. They will be the CTOs of the future, shaping the digital landscape and driving business success.
What skills do you think are most important for the modern architect? Id love to hear your thoughts in the comments below. Also please find some more interesting finds on CTO!
Human Error -> Root cause AWS S3 outage is found An authorized S3 team member using an established playbook executed a command which was intended to remove a small numb
Human Error -> Root cause AWS S3 outage is found
Martijn Veldkamp
“Strategic Technology Leader | Customer’s Virtual CTO | Salesforce Expert | Helping Businesses Drive Digital Transformation”
March 3, 2017
An authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.
The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region.
While these subsystems were being restarted, S3 was unable to service requests. Other AWS services in the US-EAST-1 Region that rely on S3 for storage, including Elastic Compute Cloud (EC2), Elastic Block Store (EBS) volumes and AWS Lambda were also impacted while the S3 APIs were unavailable.
Read the whole story here
Blockchain could solve Data Integrity problems As the world relies more heavily on data as the basis for critical decision-making, it is vital that this data can be tru
As the world relies more heavily on data as the basis for critical decision-making, it is vital that this data can be trusted. And that trust is the key issue here.
People (Data Scientists, Chief Innovation Officers) are looking for ways to automate using data. Automation translates to efficiency which translates to value. This automation trend has increased through advances in business intelligence, big data, the rise of IoT and the necessary cloud infrastructure.
So why do I raise this trust issue? Isn’t this solved perhaps by the Industry standard DMBOK? It states the possible Data Quality Management processes.
Because data is vulnerable, not just to the breaches we hear about in the news, but to a much more subtle, potentially more destructive class of attack, an attack on data integrity. Data isn’t stolen but manipulated and changed.
Like this tech-savvy Staten Island high school student who studied advanced computer programming at an elite computer camp who used his skills to hack into a secure computer system and improve his scores.
Enter the Blockchain
A possible solution for assuring data integrity could be blockchain technology.
In a blockchain, time-stamped entries are made into an immutable, linear log of events that is replicated across the network. Each discrete entry, in addition to being time-stamped, is irreversible and can have a strong identity attached. So it becomes irrefutable who made the entry, and when. These time-stamped entries are then approved by a distributed group of validators according to a previously agreed-upon rule set.
Once an entry is confirmed according to this rule set, the entry is replicated and stored by every node in the network, eliminating single points of failure and ensuring data resilience and availability.
Future
Because the promises of data integrity and security are so strong, new systems can be built to share blockchain-enforced data among organizations who may not trust each other. And once an ecosystem has shared data that everyone can trust in, new automation opportunities emerge.
Smart contracts are perhaps the next step. It makes it possible that different parties create automated processes across companies and perhaps industries. Blockchain could be an ecosystem for cross-industry workflows involving data from multiple parties. Now an entire new class of loosely coupled integration applications can be created.
Cloud adoption! Do you have a strategy? As conversations about the Cloud continues to focus on IT’s inability at adoption (or the gap between IT and Business), organiza
Reliability is often attributed as one of the reasons some organizations are wary of the cloud.
Last week, Amazon, Rackspace and IBM had to “reboot” their clouds to deal with maintenance issues with the Xen hypervisor. Details were scarce but it was pretty quickly established that an unspecified vulnerability in the Xen hypervisor was the issue.
The vulnerability, discovered by researcher Jan Beulich, concerned Xen hypervisor, open-source technology that cloud service providers use to create and run virtual machines. If exploited the vulnerability would have allowed malicious virtual machines to read data from or crash other virtual machines as well as the host server.
Not all providers had to reboot their clouds to upgrades or maintenance. Google and EMC VMware support the notion of live migration, which keeps internal changes invisible to users and avoids these Xen reboots and Microsoft uses (customized) Hyper V so they did not have that vulnerability.
It is interesting to see what “uptime” means in this context. In many reports of this nature, “uptime” doesn’t take into account “scheduled downtime.” And that could very well be the case here, as well. If one does a little bit of math:
- 99.9% uptime is 8.77 hours of downtime per year
- 99.99% uptime is 52.60 minutes of downtime per year
- 99.999% uptime is 5.26 minutes of downtime per year
Although some users complained about the outage most where complaining about (the lack of) the providers’ communications.
Cloud providers cannot be considered as a black box anymore. As an architect we need to know the limitations of the architectural components the provider uses such as Xen. We need to know how often these kinds of reboots have occurred, and how the provider handles transparent maintenance.
We also need to consider the lines of communications. Providers often drop the ball here. People are often unhappy because they didn’t get much (or any) heads-up about the reboot, not about the reboots itself.
We should remember that outages and other disruptions are few and far between these days, so these rare event get extra media attention.
Cloud adoption! Do you have a strategy? As conversations about the Cloud continues to focus on IT’s inability at adoption (or the gap between IT and Business), organiza
As conversations about the Cloud continues to focus on IT’s inability at adoption (or the gap between IT and Business), organizations outside of IT continue their cloud adoption. While many of these efforts are considered Rogue or Shadow IT efforts and are frowned upon by the IT organization, they are simply a response to a wider problem.
The IT organization needs to adopt a cloud strategy, a holistic one is even better. However, are they really ready for this approach? There are still CIOs who are resisting cloud.
A large part of the problem is that most organizations are still in a much earlier state of adoption.
Common hurdles are
- The mindset : “critical systems may not reside outside your own data center”
- Differentiation: “our applications and services are true differentiators”
- Organizational changes : “moving to cloud changes how our processes and governance models behave”
- Vendor management : “we like the current vendors and their sales representative”
In order to develop a holistic cloud strategy, it is important to follow a well-defined process. Plan Do Check Act fits just about any organization:
Assess: Provide a holistic assessment of the entire IT organization, applications and services that are business focused, not technology focused. Understand what is differentiating and what is not.
Roadmap: Use the options and recommendations from the assessment to provide a roadmap. The roadmap outlines priority and valuations .
Execute: For many, it is important to start small because of the lower risk and ramp up were possible.
Re-Assess & Adjust: As the IT organization starts down the path of execution, lessons are learned and adjustments needed. Those adjustments will span technology, organization, process and governance. Continual improvement is a key hallmark to staying in tune with the changing demands.
Today, cloud is leveraged in many ways from Software as a Service (SaaS) to Infrastructure as a Service (IaaS). However, it is most often a very fractured and disjointed approach to leveraging cloud. Yet, the very applications and services in play require that organizations consider a holistic approach in order to work most effectively.
Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices
We’re about to go back once again in the circle to decentralize and give a greater role to local storage and computing power.
It depends on the nature and the amount of data that needs to be stored and it’s process demands. With the enormous rise of the amount of data because of the ‘Internet of Things’ the nature of the data is becoming more and more diffuse. These developments lead to yet another revolution in data area: The Fog.
Smarter? Or gathering more data?
More and more devices are equipped with sensors; cars, lampposts, parking lots, windmills, solar power plants and from animals to humans. Many of these developments are currently still in the design phase, but it will not be long before we live in smart homes in smart cities and we are driving our cars by smart streets wearing our smart tech.
Everything around us is ‘getting smarter’ / gathers more data. But where is that data stored, and why? Where is all that data processed into useful information? The bandwidth of the networks we use, grows much slower than the amount of data that is send through it. This requires thinking about the reason to store data (in the cloud).
If you want to compare data from many different locations, for instance data from sensors in a parking lot via an app where the nearest free parking space is, then the cloud is a good place to process the information. But what about the data that can even better be handled locally?
Data Qualification
The more data is collected, the more important it will be to determine the nature of the data is and what needs to be done with it. We need to look at the purpose of the collected data. For example: If the data is used for ‘predictive maintenance’, which monitors something so that a timely replacement or preventive maintenance can take place, it does not always make sense to send the data to the cloud.
Another example is the data that is generated by security cameras. These typically show 99.9% of the time an image of room/space that has not changed. The interesting data is the remaining 0.1% where there is something to see. The rest can be stored locally, or even not at all. This filtering of useful and useless data calls again for local power.
This decentralization of computing power and storage is a recent trend that Cisco calls ‘fog computing’. With distributed intelligence an often more effective action can be taken in response to the collected data, and unnecessary costs of bandwidth and storage can be avoided. This is a development that goes very well with the transition to the cloud.
Cisco
Fog Computing is a paradigm that extends Cloud computing and services to the edge of the network. Similar to Cloud, Fog provides data, compute, storage, and application services to end-users. The distinguishing Fog characteristics are its proximity to end-users, its dense geographical distribution, and its support for mobility. Services are hosted at the network edge or even end devices such as set-top-boxes or access points. By doing so, Fog reduces service latency, and improves Quality of Service (QoS), resulting in superior user-experience. Fog Computing supports emerging Internet of Everything (IoE) applications that demand real-time/predictable latency (industrial automation, transportation, networks of sensors and actuators). Thanks to its wide geographical distribution the Fog paradigm is well positioned for real time big data and real time analytics. Fog supports densely distributed data collection points, hence adding a fourth axis to the often mentioned Big Data dimensions (volume, variety, and velocity).
Unlike traditional data centers, Fog devices are geographically distributed over heterogeneous platforms, spanning multiple management domains. Cisco is interested in innovative proposals that facilitate service mobility across platforms, and technologies that preserve end-user and content security and privacy across domains.
The future? It will be hybrid with foggy edges.
Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices
One thing is for certain, we will spend a good part of 2015 talking about, discussing and disagreeing on how we now need to move, deliver, transport, carry, send and integrate the various component elements that make up our Business Applications.
The advent of Cloud, virtualization and managed hosting technologies means that we have all become used to the ‘as-a-Service’ extension as we now purchase a defined selection of software applications and data that are increasingly segmented and componentized in their nature.
Because of the Cloud, businesses run on mobile devices with employees, customers and partners easily collaborating, data securely stored and accessible from anywhere in the world all without a worry about the infrastructure. That’s someone else’s problem, isn’t it? With low monthly prices, who wouldn’t sign up and embrace a SaaS app that makes your life easier.
All the convenience comes at a price.
That price is silos. Instead of tearing down silos, SaaS applications builds strong and high walls around functionality and data. Not like those traditional legacy silos but loads of little silos within and in between departments and teams. Instead of bringing teams into alignment, they are separated into fiefdoms of data if one does not govern the Cloud.
Price reductions for IaaS lead to? In the last six months the continued decline in pricing for IaaS is a signal that more business is sought. IBM thinks that the prices
One of the propositions of cloud is that it should be possible – through the use of intelligent software – to build reliable systems on top of unreliable hardware. Just like you can build reliable and affordable storage systems using RAID (Redundant Arrays of Inexpensive Disks).
One of the largest cloud providers says: “everything that can go wrong, will go wrong”.
So the hardware is unreliable, right? Mmm, no. Nowadays most large cloud providers buy very reliable simpler (purpose-optimized) equipment upstream of suppliers in the server market. Sorry Dell, HP & Lenovo there goes a large part of your market. Because when running several hundred thousands of servers a failure rate of 1 PPM versus 2 PPM (parts per million) makes a huge difference.
Up-time is further increased by thinking carefully about what exactly is important for reliability. For example: one of the big providers routinely removes the overload protection from its transformers. They prefer that occasionally a transformer costing a few thousand dollars breaks down, to regularly having whole isles loose power because a transformer manufacturer was worried about possible warranty claims.
The real question continues to be what happens to your application when something like this happens. Does it simply remain operational, does it gracefully decline to a slightly simpler, slightly slower but still usable version of itself, or does it just crash and burn? And for how long?
The cloud is not about technology or hardware, it’s about mindset and the application architecture.