Choose your "poison"

Photo by Hunter Hailey on Unsplash

Obviously you gauge if it meets your requirements, future scale or adoption effort tolerance etc.

I feel what might get missed sometimes is the architectural tradeoffs you opt into when picking one (cause the creators would have definitely handed certain tradeoffs to favour some features over others).

Some examples:

1) Policy tags on Google BigQuery columns are a different abstraction altogether, and loosely coupled with bq schema. So you need to consider accordingly when rolling out data governance/schema evolution processes on your BQ tables/data.

2) You already use NiFi for routing streaming data, perhaps to Kafka. Now there are batch processing use cases.
NiFi uses FlowFiles (a streaming abstraction) where you would have to split/merge chunks of stream to emulate batches. Also it doesn't have long term replayability (not suiting kappa architecture if you want it)

So you might choose ksql/flink to handle batches with windows and build your pipelines accordingly (both stream & batch), still orchestrated from NiFi.

Would you think of Airflow in this mix where we get a garden variety of dependency handling and operators?

Batch tasks will be handled well, but even with continuous mode it's not a true stream process orchestrator. So there you have to reason about streaming dags differently than batch ones, favouring pull based discrete actions more.

I know "throwaway architecture" is also a thing, but often infrastructure becomes the framework in these scenarios and interfaces are tightly coupled/dependent on underlying tooling/behaviour.

Eventually a system develops, utilities & developer workflows sprout in your org to support the tooling/framework and sticks.

What you would have to build/integrate to plug gaps resulting from your choice, should also matter.