Everyone wants to say that something is changing, that it is changing in some sort of seismic way, that they are the one person that understands the change, and that you need to talk to them before it is too late, or in order to get ahead of everyone else.
It’s exhausting. Also not really true. Text to SQL for example: This approach is somewhat more complex and difficult, and expensive to set up than traditional BI. Probably creates more dependency and requirements on the data engineering team, and requires them to think about the deployment differently, with new approaches and new technology. This is not faster or cheaper. But also not always better.
Once deployed it looks cool. But does it add value? I don’t know.
Currently, we deploy data models and data to analysts and “BI content creators”, who then create content. If they are in IT, they create content based on requirements, if they are in the business, “Analysts” are working with others in the business in a more ad hoc way, possibly creating content for a meeting, a decision, a set of decisions.
The IT driven approach is suitable for significant efforts that will be durable. Creating dashboards to show operational performance in customer service, outbound marketing, or manufacturing are good examples. Same with standard sales and financial reporting. The content is going to be around for a long time, there is value in them looking nice, they are going to be widely circulated, they demand a level of professional performance, capability, accuracy and reliability.
The analyst driven approach ranges from the more manual process of collecting data to load into excel, combine with multiple sources and then create content to support and share analysis, to making a dashboard in Cognos or PowerBI and using it in that environment, because all of the data needed is already available there, and the analyst is able to use those tools to complete the task. At the outset, the “requirement” in this approach is typically vague, and the nature of the analysis is exploratory. This requires considerable data and business expertise to get the information everyone wants or needs to inform decision-making.
When people struggle with the tool, and they don’t understand the data (technical: where it is, what it’s called, or functional: what it means, and how it relates to real things) AND don’t know what they want, that is a lot of friction to overcome and the process is slow. Slow and discouraging enough that many people give up and fall back into doing things that keep them busy, but do not lead to anything new and better..
Seems like for years we tried to make the tools easier to use, or at least claim we had. But, even when people are competent with the tools, but don’t understand the business, or how to translate the needs of the business into information, we’re no better off. The process of turning data into usable information is still slow and painful. People frequently blame IT for not understanding what the business wants, but it is equally true that the business did not understand how to ask IT for what it wants or needs.
This place between the business and IT is the space I have operated for 25 years and it is the unique perspective from which I see both sides of the issue.
The easiest, and least effective way to address this has been with the end user tool. And companies have thrashed about looking for the tool to solve this problem. As I mentioned earlier, people love to shop, and they love to buy solutions. It’s a lot like going to the doctor, presenting the symptoms, receiving a diagnosis, and getting a prescription. Looking at the sales of Ozempic , I would say this is a buying pattern both buyers and sellers are excited about.
Whether its Cognos, Business Objects, MicroStrategy, then Qlik, Tableau, or PowerBI, the perception of how easy these tools are to use, deploy, or manage has been a core tenet of their marketing strategies. But here we are, in the AI era, still complaining about the process of creating dashboards. Power BI has something north of 2 million youtube videos showing you how to do just about anything. Most of it is essentially useless in a business setting from an analyst’s perspective. The exotic visuals, animations and navigation aids are the glitz and glam designed to distract you from the real problem – because that’s how you sell Power BI.
This is the most common situation I see: If an analyst doesn’t feel comfortable with the tool, they download large low-level datasets into excel and then do the work there. This is very common, perhaps the standard way analysts do this work. They make a lot of errors, and over time the errors and complexity compound. When this process becomes unmanageable (errors, volume, complexity, performance, the person running this process leaves) something has to change, and for decades the real answer and what people have done are not the same.
Larger organizations attempt to address this problem with organization and structure and teams. This includes specialized analysts that gather and define opportunities and scope them into projects with defined goals, and requirements formally developed and documented. These requirements are then handed over to project managers, who then break everything down into tasks and assign them to technical team members and then manage the execution of the tasks by people with specific skills and various levels of engagement and understanding of the goals and objectives of the project, or the business. The fundamental strategy is to gain scale in delivering data and analytics to the business, but it comes at the cost of the bloat and overhead of managing those resources, and the organization reduces the effectiveness and attracts the wrong people to the team. Sometimes the organization chooses the wrong people on purpose because they are cheap.What could go wrong? This process is slow, expensive and it frustrates the people managing the process, and the business it was designed to serve. Failures and costs are a cliche.
The fundamental error here is trying to produce a custom output that is broad and hard to define, using an assembly line approach. Even in the best case, requirements are never good enough to ensure a developer with a narrow role and limited understanding of the project goal is successful. And gathering and defining requirements like this is long, difficult and expensive. And even the business doesn’t understand them when they’re done, and they supposedly represent their wants or needs. For 20 years I have screamed from the rooftop that the best way to gather requirements is interactively, and it starts with understanding how the business works, and what it is hoping to accomplish – not defining requirements for IT. That is a topic for another day, but hold that thought, because it applies to AI applications, too..
The real problem. Disjointed teams that don’t know what their teammates are doing, and they don’t care. Companies define narrow roles and skills because they want to make them commodities, place them on an org chart and then buy them at the lowest price. What person who chose to lead data and analytics as a strategic function in a business ever said I want to create a large team of people that requires layers of management and process to make them productive? The people you want in data, analytics and AI are people who want to make an impact on the business, not bureaucrats counting help desk tickets, calculating productivity and giving employee reviews.
Data and analytics talent is interested in having an impact, and being part of a winning team, growing in capabilities and value, not creating visuals on dashboards, though that will still be part of the job. For people working in the tools daily or even weekly or monthly, crafting the visuals is not the hard part, or the time consuming part, which, by the way, is also the reason that using AI in this way is not the valuable part.
Delivering data, analytics and AI is not the assembly line. Data and analytics engineers create the assembly lines, and the output is information. The assembly lines have data and tools as inputs, and information that serves the business as outputs. Though often similar, different analytic applications are not the same, and the degree to which the process of creating them can be turned into an assembly line varies. The biggest impact to improving productivity is architecture – both data and application architecture.
Data engineers do this by designing data pipelines and models to be relevant and reusable across applications. An example is creating a sales data model and pipeline at the atomic level of detail, so they can be used for sales reporting, customer and inventory analytics, forecasting and so on. This reduces complexity, and cost and creates the proverbial single version of the truth that goes in and out of fashion. On the application side, designing standard interfaces to do the universal things like provisioning, authentication and authorization – both content and data. Creating a standard provisioning and security model that addresses most, if not all these needs means the team does not have to design and operate these processes themselves, and can use standard processes managed by a team designed to do these tasks efficiently. In addition to taking these tasks off their plates, this also provides data security that enables broad deployment and leverage with content: You can use the same reports and content across the organization, controlling access to data to meet regulatory requirements, and make using data easier for everyone.
All of this is characteristic of an experienced team that is engaged, creative, ambitious, intelligent and seeking excellence. Is it an offshore team? Probably not. Is it a bunch of entry level developers with specific skills working off jira tasks, requirements documents and a project manager that doesn’t understand analytics or the project goals? No. Maybe it is a small team of highly skilled and experienced engineers using AI as part of their toolset, but AI will not do the work for you, despite what your Cursor sales rep told you.
Earlier I mentioned the things that cause friction in analytics development and engineering. Those are: difficulty with the tools, difficulty with the data, and difficulty understanding business requirements. The issues with the backroom are similar: tools, data and requirements, but there is more: architecture and engineering. The way the overall task is structured matters, the way the platform is structured matters, and the engineering and approach matters. I have been practising these things for more than 20 years, and I am pretty good at it. Doing these things poorly results in errors, performance problems, and extra cost in support, maintenance and compute.
There is a time and place for looking at tools, but the real friction isn’t the tools and it never was. It’s people, structure, and the endless parade of shopping for the “next big things” sold as shortcuts, instead of doing the work. AI won’t fix bad data, unclear requirements, or teams that don’t talk to each other. The problem has always been knowing what the business needs, how the data represents reality, and building teams that can close the gap.
That’s why “analytics engineering” matters. It brings discipline like version control, modular pipelines, and reusable data models instead of data engineering to meet requirements and Excel heroics that break under their own weight and do not offer scale and efficiency. The real leverage comes from architecture: pipelines and models designed once, used everywhere, serving forecasting, reporting, and operations without rebuilding from scratch every time.
Yet companies keep trying to replace talent and skill with software. They build big, siloed teams managed by automated tickets, workflows and templates, where no one owns the business outcome. Costs rise, timelines stretch, and the whole process serves itself instead of the business. It’s the wrong structure chasing the wrong goals. Locally optimized, globally….nothing.
And the irony? Most companies could probably get twice the output and quality with half the people, if they hired the right people and organized the teams well. Tools don’t solve this. Teams do. Skilled, aligned teams with the mandate to get the architecture right, understand the business, and deliver like engineers not like order-takers on an assembly line. AI might speed them up, but it can’t fix getting the fundamentals wrong.
At the end of the day, analytics has never been about the next tool or the next trend. It’s about building the right foundation: clean data, solid architecture, and teams that understand both the business and the technology. The tools will change, AI will add speed and capability, but none of it matters if the fundamentals are wrong. Get the structure, the talent, and the approach right, and the rest falls into place. Keep chasing shortcuts, and you’ll be having this same conversation five years from now.
Leave a Reply