Data Independence
You and your bud are jamming out to math rock at the office while you work on the MVP of Datablaster 3000 and then the senior engineer walks in to the office and says “I think we should ship without the query planner; Datablaster 3000 is for advanced users. People who know how queries are executed. We don’t have to make it easier for them to write queries.”
You nearly spit out your peppermint latte and give your friend a wary look. Is this what this guy thinks we’re doing here?
It’s understandable and easy for someone who has gone deep into the bowels of query execution and come out the other side having been inculcated to think that query planning is about ease. About making it less difficult to author a query. After all, I’ve gone through the work to understand what the query planner is doing, so perhaps I don’t need it any more. But this isn’t true! Query planning, and declarative programming is not, and has never been about the ease of authoring a query, although that can be a nice benefit in some cases.
Query planning is not good because it makes writing queries easier: it is good because it contributes to separation of concerns. It allows us to chase data independence.
The perfect dream of data management is that we describe how our data looks and all of our clients have access to it in whatever version is most appropriate for them. This kind of vision is never actually realized because clients generally need to be aware of what access patterns are well-supported via indexes or caches, but it's implicitly at the heart of any system that has a concept of a "database" that is separate from an "application." There is some notion of a coherent global state, and we are all subservient to it.
Data independence is the idea that such a global database should be able to be used without knowing how the data underlying it is stored. When we say "there is a database" and not "there is a B-tree whose key is the customer_id," we are chasing data independence: we are telling you what we offer you, not how.
This is often understood, again, incorrectly, as making things easier. "Databases are easier to interact with if you use this nice, declarative, relational model." This is sort of true, but only incidentally. It's not the point. Nobody who has ever operated a database would tell you that you don't need to think about access methods: that's a very important part of running a database! You need to know constantly what columns need to be indexed and which ones you can avoid indexing. Nobody should be arguing that declarative query models enable you to ignore the realities of database access. That's how we get really bad takes like "don't worry about the algorithm; trust the database to do the smart thing!" parroted constantly.
There's a much simpler, and better understood term for data independence: it just means "separation of concerns." By having queries express what they want rather than how it should be computed, we allow the part of our application which uses those queries to not care about the physical layout of the data. We're enabling a pattern for programming in the large (how a big piece of software can be assembled out of smaller pieces), not expressing an opinion about programming in the small (how those pieces themselves should be constructed).
The senior engineer frowns and says he'd just write a function to encapsulate which index to use; if we have the customer_id, then use the index for it, if you have organization_id, use the index for it. Senior engineer has re-invented a query planner.