IBI-073-Enterprise Data And AI
This is an opinion piece replying to Saurabh Gupta's recent survey results...
Hello and Welcome.
I am Esther.
I am Peters A I Assistant to create voice overs.
I will simply read Peters blog posts, so that you have a choice of reading the blog post, or listening to my voice.
Hello and welcome Gentlemen.
As I have recently announced on my channel.
I will be doing some opinion pieces and response videos.
In this blog post I wanted to talk about an excellent blog post Saurabh Gupta recently released.
Firstly, I highly recommend you read Saurabh’s blog post.
You can read it on the button below.
Now.
I will include quotes for the sections I am directly responding to.
In all honesty, I have been arguing most of these points now for nearly thirty years.
It was nineteen ninety six that I first argued that a data warehouse should contain all enterprise data in a dimensional model that was properly cleaned, made subject oriented, integrated and available for direct query to the business analysts inside the business.
Prior to nineteen ninety six we did not have the ability to put all data into the data warehouse.
The development of the E T L system was simply too time consuming and too expensive to possibly put all the data into the data warehouse.
It should also be noted that in that era there was no incremental extract capability from most large systems.
Many large systems were still on tape or they were tape based systems that have been migrated to disk and still operated like they were on tape.
Because of this E T L processing had to be full image dumps of operational systems and then delta detection was performed.
This was very expensive in processing and time consuming in run times.
Prior to nineteen ninety six the mapping rate of fields from source systems to target systems in dimensional data warehouses was one hundred field per programmer month or even less.
The thing that changed in nineteen ninety six was that I wrote my first E T L software.
I single handedly skyrocketed the E T L mapping rate to around one thousand fields per two hundred and twenty hour work month.
This was an unheard of speed of E T L development.
My E T L software made it possible to put all enterprise data of the day into the data warehouse.
My very first project to do exactly that was the Hong Kong Manu life subsidiary in nineteen ninety seven.
Manu life was the largest insurance company in Canada at the time.
The other thing that we were very aware of at that time was the meta data layer over the top of the dimensional models to make the data models easily understandable by business analysts.
Because I had implemented Ralph Kimballs Data Interpretation System in nineteen ninety one, I was well aware of how a Meta Data Layer can be properly placed over the top of a data model to make it easy enough for business analysts to query.
I also want to make the point at this time that when I say business analyst I mean someone with a degree on the business side of the house and not someone with an I T degree.
I T people, in the vast majority, make terrible business analysts.
A business analyst is a man who knows how his business works in great detail and lives, eats and breaths his business.
It is almost unheard of for a man with an I T degree to become even a mildly competent business analyst.
I am one of the few such men I have ever met.
Sean Kelly was one of the few others.
We are so rare we make hens teeth as common as dust on the ground.
So with that short back ground.
Please allow me to quote and then address specific comments.
Quote.
The Modern Data Report reveals how enterprise data is nowhere near A I-ready. A I cannot reason over data that humans themselves cannot rely on.
End quote.
This is the fundamental finding of the paper. Enterprise Data is not ready to be fed into A I systems.
The report then mentions some main findings. I will repeat the findings.
And then I will reply to the specific findings and comment.
I will prefix all my comments with this comment.
Any company who wants to solve all the issues mentioned below are welcome to download my now free E T L software and data models and use them to solve all the problems mentioned for free.
If you would like to have the tool I recommend you give to business analysts so they can answer their own questions then you can get a copy of the meta five single user software from meta five dot com.
Meta five is the current name for Ralph Kimballs Data Interpretation System. Meta five is a play on words because Ralph Kimballs company used to be called Metaphor. So the next generation of Metaphor is Meta five.
When I say Meta five I mean the current version of Ralph Kimballs Data Interpretation System that now runs on windows and is available in a single user edition as well as the enterprise edition.
Meta five has a basic meta data layer that is perfectly adequate to empower business analysts to ask their own questions of their own dimensional models.
Of course, it is not free but the single user version is very cheap.
I am not going to quote a price because meta five retains the right to vary prices for it’s software.
This is the software that Ralph Kimball invented so you can be sure it is worth the money being asked for.
Quote.
Enterprises have modern data stacks, but their data still can’t be trusted or activated: nearly half can’t fully rely on their data for decisions, and most say it isn’t ready for A I.
End quote.
Correct.
From my limited observations of the marketplace over the last fifteen years it appears that the so called Modern Data Stacks have been an overall negative in the world of using data to improve corporate profit over time.
Much of what I read about modern data stacks is so bad that for many years I thought people were kidding and were not really doing what they were talking about.
What is now called the modern data stack was never going to work. It is like the hype about how Hadoop was going to replace databases. The so called modern data stack is a nonsense and was always going to fail.
The so called modern data stack did not address the fundamental issues of data warehousing which are the data models and the ability to put data into the data models. The so called modern data stack sold the snake oil that some how the T portion of E T L could be ignored.
It can’t.
If you leave the T bit out of the E T L you can not rely on your data. You will have data errors. I have known this since my very first data warehouse project in nineteen ninety one.
There are many problems with enterprise systems and they have been around since day one of enterprise systems.
The three main problems are different keys of similar data in different systems, in valid data, and missing data.
They are three different problems but they occur in every enterprise.
If you want to have data you can give to end users in an enterprise you have to deliver that data in a dimensional model and it has to be properly prepared and cleaned.
That we are in twenty, twenty six and this is still the single most severe problem in enterprises using their data is a testament to the result of our industry refusing to take my advice for more than twenty five years now.
Quote.
The real work of data teams is no longer analysis. It’s search, reconciliation, and validation. Finding and confirming data consumes more time than actually using it.
End quote.
Correct.
Because our industry has not taken my advice of put all your enterprise data into your data warehouse and maintain it properly the so called data teams are endlessly searching around for data to answer questions that arise in the business. Then they do E T L on that specific set of data that has now been asked for. Rarely are they putting it into an enterprise data warehouse. They treat it as a one off. Until the next time someone else asks for it.
If you do your data acquisition on demand then you will spend for ever doing data acquisition and cleaning.
The only way to build an enterprise data warehouse properly is to set out from the beginning to place all enterprise data into the data warehouse. To do this you need a very large and comprehensive base data model to start out with.
If your consultants are selling you the idea that it is best to start with a blank piece of paper for your enterprise data warehouse data model? Then please fire them and save yourself a lot of problems.
As I said above. I have been recommending that enterprises put all their data into their data warehouse since nineteen ninety six. Some have taken my advice. Most have left some data out that they have felt they will never need. Almost all of them have regretted that and added that data later.
If your consultants are saying.
Quote.
Start small and just put the most important and most urgent data into the data warehouse and we will figure the rest out later.
End Quote.
Please fire them and save yourself a lot of problems.
In the few accounts I have visibility in to now they consistently have the problem of adding new data because they did not plan to add all data from day one.
Quote.
Context is fragmented across tools, breaking meaning, lineage, and shared definitions: data exists, but the knowledge required to interpret and act on it does not travel with it.
End Quote.
Correct.
In virtually every account I have ever worked in the E T L tool requires it’s own dictionary and the various front end tools all require their own dictionaries.
Further, the data lineage has been appallingly documented in almost every customer I have ever seen that is not mine.
In my very first project in nineteen ninety one the most asked question was.
Quote.
How was this field calculated.
End quote.
In nineteen ninety three I was doing a pilot at the Saint George Bank in Australia. Because of our pilot it was discovered that the banks liquidity ratio was being incorrectly calculated and the bank was in violation of the existing legislation as to the liquidity ratio that banks must observe.
Of course, the Australian government is not really going to shut down a bank for violating their liquidity ratio requirements. But it was considered so bad that the system was shut down because no data is better than wrong data.
Given that experience I realised just how important it was to be able to know and understand how a field was calculated. And yet here we are today and the most asked question in the accounts I talk to is.
Quote.
How is that field calculated exactly?
End Quote.
Of course, because the front end tools have a separate dictionary to the E T L software there is never an understandable link between the front end tool and the data model and then back through the E T L.
My old friend Ernie Ostick from Ardent has designed a wonderful piece of software called Manta that is now the worlds best software to do exactly what I have described here. If you want the best meta data management tool in the world just contact Manta. The price is eye watering but it’s worth it for big companies.
Quote.
A I adoption is blocked by weak data foundations, not model capability. Without semantic layers, standardised metrics, and trust signals, A I cannot reliably trigger decisions or close feedback loops.
End quote.
Correct.
If your data warehouse can not deliver reliable data, then sending that data into an A I is a case of garbage in garbage out.
Your A I will give you answers. And the answers will even seem reasonable. But they will be wrong.
Example. I once did a telco where we replaced Business Objects reports written based on an Oracle Replica of their billing system. There were eighty five Business Objects reports we were replacing using our data warehouse.
Of those eighty five reports precisely NONE of them were correct when written against the replica of their billing system. They were not even able to get one in eighty five correct.
That is how hard it is to get the right answer out of an enterprise level operational system even if you are an expert in it.
A I can not be made useful when it is fed data that is incorrect. And the worst kind of incorrect data is data where the level of incorrectness is so small that it is not obvious and can only be found by comparing it to a properly built dimensional data warehouse.
Quote.
Practitioners are aligned on the fix. Convergence over complexity: unified access, embedded governance, shared semantics, and self-service are the foundations required to close the data activation gap.
End quote.
I could not have said that better myself. In fact I have been saying this for thirty years!
Whether it be Business Intelligence or Artificial Intelligence.
If you want to use the data in your data warehouse you absolutely need to do the same things in the data preparation.
You need to deliver a data model that is simple enough for a business analyst to read himself. If a business analyst can not read and understand the data model then no A I can read it or understand it either.
If your data warehouse data model is too complex for a skilled business analyst to query it himself? Then your AI will not be able to read it either.
I will get to unified access in a minute.
With embedded governance this is critical. Your E T L system should make sure that all the data in your data warehouse balances. It should make sure that records that are suspect are marked invalid in the staging area and not sent into the data warehouse.
Where you are implementing finance systems like the General Ledger or other ledger systems they must balance to the cent. The E T L system should have processes in place to balance the ledgers. It should be able to detect one single cent missing.
There should be one set of definitions for all fields that are published and common. That set of definitions should then be used to feed into the different front end tools that are demanded to be used by the business users.
In this way when the question how was this field calculated arises in whatever tool the business user is using? The time to answer that question is minimal.
Because all front end tools have their own dictionaries you can not expect them to use the one central dictionary as many people claim to hope to do. But you can have them refer to the one central dictionary and then the extra cost of using those front end tools is easier to understand.
Now I want to comment on unified access and self service.
Ralph Kimball and his colleagues invented self service access to data at Metaphor Computer Systems in the early nineteen eighties. To this day, more than forty years later, I have not seen any tool that compares with Ralph Kimballs Data Interpretation System for it’s ability to empower business analysts to analyse their own data.
Sadly, I T has sold the snake oil that what business analysts need is endless dashboards to analyse their data. This has always been a lie.
The vendors had a financial interest in selling dashboards. They got the license revenue per dashboard.
The I T department had a financial interest in selling dashboards. They got the salaries for creating these mostly useless dashboards.
The people who did not have an interest in the creation of endless dashboards were the shareholders, the customers and the non I T employees of the companies that built these endless dashboards.
For thirty five years I have shouted out that the best way to give self service to business analysts is Ralph Kimballs Data Interpretation System.
But not even Ralph would tell people about his own software because it was owned by I B M by then.
I would like to make my position clear.
The single best way to give self service access to data to business analysts is using Ralph Kimballs Data Interpretation System. You can buy single user versions for what can be considered a pittance compared to the return on the investment of using it.
Next I would like to talk about unified access.
This will be more controversial. But it’s about time someone said it.
There is only one piece of business software on the face of this planet that can be truly called universal.
That software is called Excel.
Whether you love Excel or you hate Excel.
You can not argue that it is the single most widely used piece of software in the business world.
I would make the argument that if you want universal and unified access to data.
Then the way to do that is via Excel as much as possible.
Certainly there are cases where the volumes of data to be reported on exceed the approximately one million row limit in Excel. No one disputes that.
However, there is almost no report that is useful to a business analyst that contains more than one hundred thousand rows of data.
I would even go so far as to say that ninety nine point nine nine percent of all reports and dashboards that are of any earthly use to business analysts have less than one hundred thousand rows of data in the actual report.
Yes, the business analyst may need to look at more data underneath the report in the data warehouse during the exploration process.
But I posit that there is almost no report a business analyst needs to answer questions that require more than one hundred thousand rows of data in the report.
I would propose, posit, argue, that the world of A I and Business Intelligence would be greatly improved if the default tool used was Excel.
I would propose, posit, argue, that other tools were only used after Excel was ruled out because the report needs more than one hundred thousand rows or required functions that Excel can not implement.
For example MicroStrategy, now called Strategy, scales to levels that Excel simply can not scale to. You can roll out a Strategy implementation to more than one hundred thousand users. And it can then give interactive access to the underlying data warehouse.
You can’t do that with Excel.
You can make the argument that you can go some way to that with the At Scale product. But At Scale comes with a very significant price tag that only large enterprises are going to consider.
I am prepared to go on the public record saying that for universal and unified access to data, the tool that is at the top of that list is Excel.
I am prepared to say that unless Excel can not deliver what is needed then it should be very strongly considered.
Because I have made such a controversial statement please allow me to follow it up with this.
With meta five it is now possible to store the S Q L to be used in a report in the cloud. Meta five can read that S Q L from the cloud into the single user desktop. It can then apply parameters to the S Q L and then send the S Q L into an O D B C connection to read the data.
For example.
If you are a retailer and you have twenty region managers who all have a set of daily reports they receive that are in Excel.
It is possible to use Meta five to generate twenty sets of their daily Excel reports that contain just the data for each of the region managers. The data is embedded into the Excel workbooks.
The region managers can then read those reports on disconnected remote devices as they wish. There is no round trip back to any data warehouse to read the report or to click on the Excel filters.
Of course, if the design point is to put the data into the Excel workbook, then there is processing time to do that. This becomes the limiting factor for using Excel with embedded data to produce reports and dashboard.
Everyone is well aware of this limiting factor.
When this processing takes too long then Excel can be made to make the round trip to a central data warehouse to query data.
Quote.
Almost seventy percent of respondents say their data isn’t clean or trustworthy enough for A I.
Sixty five percent say their data lacks the clarity and business context A I needs to be useful.
End quote.
Correct.
This is a repeat of much of what has been said above. If you do not use dimensional models and if you do not have proper E T L you will have a case of garbage in and garbage out for A I usage of your enterprise data.
I would go so far as to argue that in order for data to have clarity and context and to be in a form that can be queried.
Then it is necessary to deliver the data to the business analyst or A I in a dimensional model.
I built my first dimensional model as a prototype in nineteen ninety three. I made some serious mistakes.
I built my first proper dimensional model in nineteen ninety four and five. Once I had it right and it worked properly I knew and understood why a multi level dimensional model was by far the best data model to give to a business analyst.
Michael Saylor at Strategy figured out the same thing at about the same time. He committed Strategy to only being able to read multi level dimensional models. Today Strategy reigns supreme at the top of the Business Intelligence Tools stack because of that decision.
Just as a side note. It is now possible to read Strategy multi level models in Power BI. Please ask me in a D M if you would like that link.
Quote.
Eighty percent of respondents rank a semantic layer with standardized definitions as the most important enabler of A I: above A I tools themselves and above faster processing.
End quote.
Correct.
If you are going to give access to data to a business analyst or an A I?
Then you need some form of semantic layer that standardise definitions of data. It’s hard to believe we are in twenty twenty six and we still have to say this.
Ralph Kimballs Data Interpretation System contains a very simple semantic layer. This semantic layer makes it very simple for business analysts or A I to read and understand what the data underneath the semantic layer means.
This is so simple that it can be used as a starting point for your own development for your own company.
If you want to standardise on the Meta five meta data layer that would be a good start if you have no meta data layer today.
Quote.
Most of the respondents suggest that discovery is where this gap first becomes visible. Eighty nine percent of respondents say finding the right data is among their top three time drains.
End quote.
Correct.
All the way back in the nineties I had a slide for Ralph Kimballs Data Interpretation System.
The slide said that without D I S the business analyst would spend seventy five percent of his time searching for, cleaning, reconciling and gathering data. He would then spend twenty five percent of his time doing actual useful data analysis.
The slide then said that with D I S the business analyst would spend twenty five percent of his time searching for, cleaning, reconciling and gathering data. He would then spend seventy five percent of his time doing actual useful data analysis.
The slide pointed out that D I S would triple the productivity of some of the most valuable people in the company.
Thirty years later we have a survey that finds that eighty nine percent of respondents are saying what we knew in the early nineties. Finding the right data and collecting it is a big time drain. A problem where we knew how to triple productivity more than thirty years ago.
This is an example of why I am so disappointed with the men in my industry sector. We have known how to solve these problems for more than three decades. And yet here we are. We can read a survey where fully eighty nine percent of the respondents say they have a problem that we knew how to solve thirty years ago.
And the reason these respondents have this problem is that my peers in my industry refused to take my advice. The men in my industry chose to listen to the snake oil salesmen at the vendors rather than listen to me.
Oh well. Eh?
Quote.
Ninety three percent of respondents say they encounter conflicting metrics. Nearly half do not fully trust their own data, and sixty eight percent explicitly state that it is not trustworthy enough for A I.
End quote.
Indeed, not only is their data not trustworthy enough for A I. It is not trustworthy enough to be used in general.
Here is a short story. This really happened. I am not making this up.
In Australia in the nineties there was an insurance company called A A M I. All Australians know this company. A friend of mine won their data warehouse project in nineteen ninety four.
I tried to give him advice and help him out to make sure his project was successful. I did this just to help him be successful and to make data warehousing more successful in Australia.
He lived in Melbourne and I lived in Sydney so it was not like I wanted to steal his client.
The project went on for two years and what my friend did was build four separate data marts feeding data from the many different insurance systems.
He had to do this because A A M I had many operational systems for their different business areas.
Of course, the numbers did not add up and the project was deemed a terrible failure.
By this time I had moved to Hitachi Data Systems and A A M I had a mainframe computer from us.
So I went into A A M I and I was trying to help my friend and help A A M I salvage their two years of investment.
But being from an I T vendor I was not allowed to speak to the business managers.
After a full year of going to talk to the D B A Manager at A A M I about their now failed data warehouse project he finally gave me permission to speak to the director of marketing and two of his senior business analysts.
What we were proposing was to build a single integrated data warehouse.
The data warehouse would be integrated, subject oriented, and historical.
From the data warehouse we would then feed the four data marts so that all the numbers added up and they could have confidence in their data.
I did a demo and it was very well accepted by the two business analysts.
The director of marketing left after five minutes. He had zero interest in what he called technical mumbo jumbo.
The demo was from eleven A M to midday.
After the demo the sales representative and I drove back to the Hitachi Office.
Our office was actually in the same street and only about two hundred meters from the A A M I head office.
The Hitachi Branch Manager was standing on the curb and signalled us to come to his office.
When we got into his office he was furious. He said that he had just had the Managing Director of A A M I on the phone demanding that he come to his office at one P M and he was very unhappy with us. So what the hell had we done?
I explained what happened and the branch manager said that I should come with him to the meeting.
We walked into the office of the Managing Director and we were not even offered a seat.
He literally yelled at us for a full five minutes about how I T vendors were to stay the hell away from his business people.
After he finished I asked him if I may ask him a question.
He said yes.
I asked him to please tell me what he would estimate the return on investment has been for the four data marts that had been built. Meaning what new profit had been created that could not possibly have been created without them.
He said zero.
I asked him if he would please be kind enough to tell me why the return on investment was zero.
He said that the numbers from the different data marts do not add up so the management team does not trust the data and does not use the data marts in their management decision making process.
I replied with words to the effect.
Yes. That is true. I know that. I warned my friend you hired to do this job three years ago that this would happen if he did not take my advice.
I have been coming here for over a year trying to save this investment and to help you make more profit.
The way you can save this investment and make more profit is to have a single, integrated, audited, validated, balanced data warehouse that feeds your data marts so you can trust the data.
He looked at me hard and asked how much is that going to cost me.
I told him it would cost three hundred thousand Australian dollars.
He yelled at me.
Quote.
And not a penny more.
End quote.
Then he said to the Branch Manager I want the contracts on my desk by Friday for sign off.
I want your guys to get started as soon as they can and to make this project a priority for your company.
I want this data warehouse thing as fast as you can possibly deliver it.
Now get out of my office and don’t be bothering my business people again!
When we got back into the car the branch manager asked me what the hell happened in there.
I said he saw a professional salesman at work.
I said A A M I has a problem.
They need good information to better manage their business.
They have spent nearly three years and over two million dollars to get the wrong numbers that are worthless.
I offered to save the two million dollar investment and offered to give them the right numbers so they can make more money.
And I offered it for three hundred thousand dollars.
And you saw, the managing director bought that deal.
For anyone reading this post.
I can assure you that the Managing Directors and Chief Executive Officers of companies want the right numbers on their reports in their senior management meetings.
When they are trying to decide what projects to approve and what new budgets to approve?
They rely on the numbers in the business cases presented to them.
If they suspect the numbers in the business case are not one hundred percent accurate they will not approve the business case. This is normal.
To get a business case approved by the senior management team for any significant amount of money for any significant project you have to have the correct numbers.
It only takes presenting incorrect numbers once to destroy your credibility and end your career at that company.
This is how important it is to make sure the data warehouse is integrated, audited, validated and balanced to the operational systems.
It is the same with A I.
Senior business managers are not going to believe the outputs from an A I if they do not believe the inputs.
It’s that simple.
Now that I have repeated that story from nineteen ninety seven it is worth repeating this quote knowing that it is fully twenty nine years later.
Quote.
Ninety three percent of respondents say they encounter conflicting metrics. Nearly half do not fully trust their own data, and sixty eight percent explicitly state that it is not trustworthy enough for A I.
End quote.
It is very sad that after twenty nine years men are still building data warehouses where fully sixty eight percent of the respondents to a survey say that their data is not trust worthy.
Of course, not all sixty eight percent will have a data warehouse.
But I would bet more than eighty percent of the respondents have a data warehouse of some sort or other and that they don’t trust their data.
I have many war stories about data not being trustworthy.
The Barings Bank debacle in nineteen ninety five is what happens when you don’t have a data warehouse doing fraud detection.
Quote.
We asked practitioners what would improve most with faster, more reliable access to high-quality data. The responses show that speed improves first, cited by eighty seven percent , followed closely by confidence at seventy eight percent.
End quote.
Correct.
As I noted above. If business analysts are given self service access to the data warehouse using meta five then their speed of data analysis will vastly speed up. This is so well understood that eighty seven percent of respondents to the survey said they felt they would speed up their data analysis if they just had the right tools, not even knowing what the right tools are.
Of course, faster, more reliable, self service access to data will also increase confidence in the data because it’s easier to find the errors.
To give you a real example. When using Ralph Kimballs Data Interpretation System the time it takes to answer any individual question is around five minutes and often faster than that.
A business analyst can simply walk up to a P C and get the answer to virtually any question in five minutes or less.
What one of my customers did was put a P C in the board room.
At the quarterly board meetings the operator of that P C was to answer any questions asked by any board member during the board meeting.
He was expected to answer any question inside five minutes.
The result was that when business cases were being proposed and a board member asked a question that the proposer of the project did not know the answer to.
Then the question went to the operator of the D I S computer and he was to answer the question inside five minutes.
The net result was that the board was able to approve, or dis-approve, business cases in the one board meeting.
This was opposed to having the person come back with the answers to questions and the business case in three months time.
The net result was that the productivity of the board meetings more than doubled. They could reliably hear more than two times the number of business cases than had been previously possible.
As a result the best business cases floated to the top and were approved. In the past a good business case might be lost simply because the presenter did not have the answers to some questions asked by board members.
The result showed up in the company results. The company I am talking about made vast improvements in their profitability in the period that this business analyst was sitting in board meetings answering the questions that came up.
When he left the company this practice was stopped.
It was no surprise to me that the companies business results went back to what they had been four years earlier.
Quite amazing really.
In Summary.
I would just like to summarise what I have said in reply to Saurabh’s original post as an opinion piece.
Saurabh has conducted a survey that appears quite well run and quite detailed.
We do not have the actual survey questions and the list of respondents.
However, I believe we can trust the findings of this survey because they almost perfectly match my experience.
Indeed, I am of the opinion that the findings of this survey here in twenty twenty six are something of an indictment on the men in our industry.
The men who are my peers.
That our industry segment is having the same problems that we solved thirty years ago is an indictment, not just an accident or anything else.
The first area is data models.
That our industry segment has not widely adopted industry standard data models for data warehouses is an indictment on our industry.
That there are still men in our industry segment who say that a company in a standard industry segment should start with a blank sheet of paper for the data model is an indictment that they are still in our industry.
In the zeros it was arguable that there were still men saying a data warehousing project should start with a blank sheet of paper.
This is because the starting price for an industry standard model was one hundred thousand euros.
Now that I have made a base industry standard model available for free?
It’s an indictment on the men in our segment that they are not using it because they want to make more money developing their own models. Basically they are gouging their customers and giving us all a bad name.
The part eye am going to play in fixing the area of data models is the creation of mega models.
I am working on two mega models with partners.
There are plenty of other large operational systems where the very best men in our industry could build a mega model.
My time is limited and I will only partner with one company per mega model.
So those who wish to get into the mega model business?
They are best advised to pick the mega model they would like to develop and then ask for my help.
You will have to make your business case to me as to why you believe you will be successful in building your mega model for your large operational system.
I will only help those men who are very well qualified to go into the mega model business.
The second area is E T L.
For my sins I am supporting some S S I S packages.
It is insane how long it takes to make changes to such packages.
For those of you who do not know.
I am expert in Data Stage and Informatica.
I have written a great deal about E T L over the years.
The simple facts of the matter are now as follows.
In nineteen ninety six I invented a way to write E T L at the development rate of approximately one thousand fields mapped from source to target in a two hundred and twenty hour work month.
I rewrote my cobol E T L into see plus plus in two thousand and two at the suggestion of Ralph Kimball.
This is the software now known as see T L.
See T L has evolved a great deal over the last twenty four years.
From nineteen ninety six until two thousand and seventeen the fastest way to develop E T L was with my software.
The rate at which E T L could be developed was one thousand fields mapped per two hundred and twenty hour work month.
We used to develop E T L using my software and then at the end of the project migrate it to whatever the client wanted to install.
We have such migration capabilities for both Informatica and Data Stage.
The migration from see T L to either Informatica or Data Stage at the end of a project took two weeks work.
And yet the men in our industry segment have not used see T L and thereby over charged their customers for E T L development.
Today the free version of see T L can be used by a highly skilled data warehouse architect to map between six thousand and eight thousand fields per two hundred and twenty hour work month.
This is more than five times the productivity of the next best tools such as Informatica and Data Stage.
Men in our industry segment are not willing to use the free version of see T L and then migrate it to whatever tool they want to go live with.
Thus over charging their customers.
This is again an indictment of those men in our industry segment.
The paid for versions of see T L are now hitting productivity levels of twelve thousand to fifteen thousand fields per work month.
I have had very long working days where I have mapped more than one thousand fields in a single working day.
My point is that in our industry segment men are still using tools like Informatica and Data Stage and writing E T L from scratch.
Using these tools provides a productivity level of less than one thousand fields mapped per work month.
They are doing this when the leading edge peak productivity is now up to fifteen thousand fields per work month.
Those men are gouging their customers and they do us all the dis-favour of harming all our reputations.
Lastly, I want to comment on development of front ends.
Since I sold the first copy of Ralph Kimballs Data Interpretation System in Australia in nineteen ninety one?
I have said that the most appropriate person to perform data exploration and data analysis is one of the five smartest business analysts in any large organisation.
I have said that I T people have no place performing data analysis because I T people have no clue how businesses work.
The I T industry has done the exact opposite of my recommendations so as to line their own pockets for useless work.
We are now reaping what we have sewn.
I T people are widely considered a joke on the business side of the house.
I was recently talking with the C E O of a company.
She asked me directly.
Quote.
Why are I T people so arrogant when they know so little about business?
End quote.
She genuinely wanted to know how someone who did not know the difference between a debit and a credit could be arrogant about their knowledge of business.
So I explained that to her.
I will say it here again in twenty twenty six.
I T people have no place in performing data analysis.
After more than thirty five years in business intelligence it is my considered opinion that the only people who should be performing data exploration and answering business questions are business analysts.
Primarily it should be men with business degrees who live, eat and breath their business.
I would also argue that no dashboards should be created by I T until the business analyst gives them a working prototype in Excel to show the I T people what to do.
In the last thirty years my experience has been that the vast majority of dashboards I have seen developed, more than ninety percent, have not been used for any significant period of time.
They fall into dis-use very quickly.
My experience has been that in the vast majority of cases the idea of the dashboard has been pushed to the business by the I T shop.
My experience has been that the I T shop has pushed the business into having dashboards to answer lots of business questions.
My experience has been that a single meta five P C would have provided a far better solution.
This is because the business analyst could have answered his own questions without the need to get in the I T queue and would not need to explain his question to anyone in I T.
After all this time I am more convinced than ever that the best way for a large enterprise to sustainably increase profitability over the long term is to give the five smartest business analysts a copy of Meta five and access to as much data as possible.
My opinion is that the data should be in a dimensional data warehouse and that the meta data layer provided can just be the simple Meta five dictionary. If there is more meta data available it can be added to the see T L dictionary manually.
My opinion is that in order to detect out of the ordinary situations meta five jobs can simply be scheduled to go looking for those unusual situations and can report on them using Excel and emails.
If you wanted to use A I to do the same scanning and reporting?
Then that would be fine too.
However it is cheaper and easier to do in Meta five.
This is where I think we are at.
And if my industry segment took my advice in this post?
Perhaps in five years time when Saurabh performs a similar survey some of the numbers will be far lower than they are today.
And with that?
I hope you found this blog post interesting and informative.
Thank you very much for your time and attention.
I really appreciate that.
Best Regards.
Esther.
Peters A I Assistant.