(date unknown but notes from years ago and then written somewhere between 2022-12-07 and 2023-03-06)

When evaluating a vendor one of the core requirements that I have is that the entire state on the vendor’s side can be accessed and modifierd through an API. Vendors are eager to say “yes, absolutely, our entire UI is built on top of our API” (which sometimes, unbeknownst to them, serves as a counter argument: some UIs are that terrible). But that’s just the start and what I need is not just that everything is available through their API, but that the API also makes reasonable guarantees on its correctness, especially when my own application’s correctness depends on it (there are multiple levels of how much I care about it - more on that some other time).

To put it in terms that maybe all of us in software development can understand: when integrating with an API, I need guarantees at least as similar to the ones I already have in any decent database. For the correct functioning of systems on which I work, I often have to treat the API as if it were another of my databases because the state behind vendor’s API has to seamlessly integrate with the state on my end.

As an example, consider what would you think of this database:

1. This db doesn’t provide me with any locking mechanism, pessimisti or optimistic. There is literally no exposed concept of a transaction.
2. The data modification operations are not atomic e.g. if you INSERT a set of records, the set may end up being partially committed.
3. The state is only eventually consistent so you cannot query to understand what was committed and what was left out. But even when I can query it, there is no way to retrieve the complete relevant state in a single call.
4. Db treats idempotent operations as errors.
5. Db is throttling you.

You would never consider using such a database! And yet, that is the sorry state of some vendor APIs.

The workarounds

Some of the above issues have workarounds, others are simply there as a reminder that the vendor doesn’t care about the API. The strict minimum to keep usefulness and provide a better quality of service to my customers is:

1. To work around lack of transactions, I have to implement an additional queuing mechanism to the component that interfaces with the vendor’s API (rule is: only one component per vendor, hopefully behind a non-leaky abstraction). No configuration processes may ever be allowed to run concurrently for the same configuration so now my system is more expensive to build, maintain and run.
2. To avoid having broken configurations, all new versions have to be done from scratch rather than just by incrementally changing them. This both increases the duration of the configuration process, once again, and increases its fragility: I cannot retry a single failed call as the state may or may not have been half committed so I have to redo the whole configuration process. And recall that each can involve hundreds of separate API calls (which may also be throttled!)

Switching away

So why not switch to another vendor? Without overindexing on the example, there are in my mind three main reason to not switch from a vendor with such a bad API game:

1. Their core offering is sufficiently differentiated for my use case. My (weak) impression is that vendores that are so differentiated often know it and just don’t care enough to improve the APIs.
2. The effort to switch away is too great for the current software investment horizon: if I have managed to get to a reasonably reliable quality of service on top of the vendor, I will prefer to provide new value to my customers.
3. When switching away you replace a set of known and already quantified issues with a completely new set of issues albeit unknown and not yet quantified. In general the technologist overpromise and sales people overpromise on top of those promises. Sometimes you just cannot de-risk everything in advance, due to lack of time, resources or simply because of the blind spots (I would never have thought to check if API is atomic within a single call)

API configuration managment for vendors

If you are a vendor with a user-facing API, please consider the following requirements:

1. Provide atomic configuration managment and strive to minimize the number of calls that your customers need to make. For better or for worse, yours is one of the databases with which my system has to integrate and you should give me the tools that we all expect from any half-decent database.
2. Make full state management available through API. I can think of the reasons not to make everything available (e.g. an interactive feature that may be expensive to run if allowed to run automatically) but certainly all the actual state that dictates the behavior of the service must be available (which once again shows that state managment API is just there to ensure correct state in the database)
3. Be kind: if in your configuration deleting something that doesn’t exists will not lead to an actual misconfiguration, help me out and don’t return an error. If inserting something that already exists is idempotent, again, help me out, don’t return an error (I realize this one requires more work)
4. Earnestly integrate with Terraform and the likes. There are changes that cannot be declaratively expressed in a single pass but mutliple passes are entirely acceptable as long as they are atomic (right?)

Last modified on 2023-03-05