I could not have said it better myself. From what I can see AI projects are failing because the data that it is using is not fit for purpose.

SAP is as much as admitting this with this purchase despite having SAP BW.

I have been hearing for a while now that AI projects are failing because the underly enterprise data is not fit for purpose. It does not surprise me at all.

From the press release.

Most enterprise AI projects fail to deliver value not because of the AI itself, but because the underlying data is fragmented, locked in proprietary formats and stripped of the business context that makes it meaningful. The result is a familiar and costly pattern: pilots that cannot scale, slow integration of new data sources, duplicated engineering work and compliance risk when organizations cannot explain how an AI-driven decision was reached. Dremio helps eliminate that data fragmentation and integration friction. The acquisition will complement the SAP Business Data Cloud and SAP HANA Cloud offerings to ensure seamless data integration across SAP and non-SAP data with high performance and low cost to accelerate AI-ready context and time-to-value for AI.

“Enterprise AI doesn’t stall because the models aren’t good enough; it stalls because the data isn’t ready for AI agents,” said Philipp Herzig, CTO, SAP SE. ” Dremio eliminates that bottleneck. Combined with SAP Business Data Cloud, we can now take customers from raw, fragmented data to governed, AI-ready intelligence on a single open platform.”

SAP Press Release

Dremio Press Release

Share Peter Andrew Nolan

Subscribe now

And it’s also a clear “attention grabbing post”! LOL!

that you allow other women to act in public like this reflects on all of you.

That you women always refuse to answer our questions reflects on all of you.

You women complain:

“Men do not take us women seriously”.

No.

We don’t.

Here is why.

Best Regards

Peter Andrew Nolan.

I really appreciate you for doing that.

I wanted to wish you and all your loved ones a Very Merry Christmas and a Happy New Year.

I wish you and all your loved ones the very best for the New Year.

Best Regards

Pete

I was talking another man here earlier today. He related to me his health issues. So I thought I would put up a post for CyberScan International.

I first came across the prior generation of this machine in 1996. I was very sick in the 1993-1995 period. I was getting progressively more and more sick. I went to all the specialists I could and they all said they could not figure out what was wrong with me.

By mid 1995 I was so sick that I was going unconscious with just 2-3 seconds notice to get myself down on the ground so I did not hit my head when I went unconscious. I was also going unconscious in meetings. When I was driving I found I had to eat sugary lollies to stay awake.

I was tested with the prior version of this machine in February 1996. The operator determined that I only had 10% of the normal level of acid in my stomach.

I was, literally, starving to death.

Over the years the machine evolved into this current edition.

I was treated with this current edition of the machine in 2011. With just three visits my health was transformed back to feeling like I did when I was a 14 year old.

So, you have my word that this machine works. Whether the operator knows how to use it or not is another question, and so you want to find a reliable operator. Or you can buy one for yourself and your family and friends. It only takes a week or two of study to learn how to use it effectively. I guess you get better over time.

The way the current version works is that you put your hand on the panel of the machine and it reads the signals of your body in about one minute. Then, with the operator, you go through and decide what the highest priority issues you want to deal with are.

This process can take up to half an hour depending on how ill you are, or, more correctly, how unbalanced your body is. The operator puts instructions into the machine to set the machine for the next step.

The next step is to put a bottle of special water on the machine. The machine then programs the water which will signal to your immune systems what you want to improve.

Then, each day, three times a day, you put three drops of this water under your tongue. I think that goes for 2-3 weeks from memory. The programmed water signals your immune system and your immune system then starts adapting and making you healthier.

The whole process looks like something you would expect to see in star trek, not in normal medicine.

If you would like to read about the machine and then find someone in your area who is a practitioner?

You can just click on the button below.

And yes, I am giving you my word this works and it is in no way a “scam”.

If you can find a practitioner in your area then you have the chance to vastly improve your health.

Since February 1996 when I was first treated with this predecessor of this machine? I have been to a doctor twice. Once was the result of being poisoned with a bacteria in Manila so you can’t blame the machine for that.

The other time was for a bacteria infection in my ear from the very warm bacteria laden water I was swimming in when I was in Saudi Arabia. The pool water temperature was 30C and it was like a soup! I just needed some anti-biotics for my ear infection.

That is 30 years of exceptionally good health.

CyberScan Site

Best Regards

Peter Andrew Nolan

1. ACDC - whom you all know.

2. Rose Tattoo.

3. The Angels.

Of course, I lived in a small town and none of these bands ever came to my home town. “Johnny Farnham” came to my home town once and we listened to him practice one day instead of doing our cricket training.

But none of the BIG bands ever came to my small home town.

Then, in 1980, I was in love with a wonderful girl named Linda.

I remember arriving at her home one time for the weekend and she had this “sneaky” look on her face. She was coy for hours and I was asking what was going on. She would not tell me.

Eventually, she asked me: “Would you like to go and see the Angels in concert"?”

Of course, I said yes, I would love to see them in concert but they only play live in the capital cities and we are not going to Melbourne any time soon.

And then…she pulled out two tickets and gave them to me.

“Angels in Albury” they were marked.

I couldn’t believe it!!

The love of my life had some how gotten tickets to an Angels concert in Albury, her home town.

I was 16 and this was the first “rock and roll” concert I had ever seen live.

To say I was “blown away” would be the understatement of the millenium.

The video above is pretty much the concert we saw that night.

The live show did not include “Dawn Is Breaking” which is on the video at 35 minutes.

But every other song on this video was on the set that night.

I can tell you.

When you go and see a show like this at 16?!

It is a life changing event.

I hope you watch this video and love it as much as I do.

And in case you don’t know…just wait until the climax of the final song.

The Angela always played the same final song in their concerts.

In my own personal opinion?

It is the greatest closing set song ever…

And I have seen MANY of the greatest bands in the world.

The Angels Final Song and end of set antics beat even Pink Floyd and the Rolling Stones.

Hello and Welcome.

I am Esther.

I am Peters A I Assistant to create voice overs.

I will simply read Peters blog posts, so that you have a choice of reading the blog post, or listening to my voice.

Hello and welcome Gentlemen.

I thought I would create a blog post with a summary of my resume in it so that anyone who does not know me can check out this online resume.

I will present this resume as a narrative of my experience rather than as a normal resume.

This is because this is more normal for videos.

You can get a copy of my client resume from my freebies document.

That is a more normal resume you would use for reviewing a persons professional career.

Most people who have been in data warehousing more than twenty years already know me.

For the last fifteen years I have not been such a public figure due to my woke cancellation for defending the rights of men and boys.

Pretty much any man who stands up for the rights of men and boys has had his life destroyed by lies and slander over the last thirty years or so.

All our ladies have supported the lies and slander of men who defended men and boys.

Personally, I think that is sad that our ladies did that. But they did.

Now that Donald Trump is in the white house and has declared that the US Government will return to hiring on merit?

I think there is a good chance that men will decide that it is time to put and end to woke cancellations.

We will see if this turns out to be the case.

As a result of all this I am stepping back into the public domain to promote my free software.

We will see if men want to use my software for free.

So.

I will start this resume narrative in nineteen eighty nine when I was twenty five.

I was the System Architect for a four million US dollar four hundred work month project that was being developed in the international development laboratory of IBM Australia.

I was the youngest System Architect ever in IBM Australia.

Many men much older than me protested my promotion because they felt they were better qualified for the job.

Many of them had a very good case as being better qualified as well.

At that time the next youngest System Architect in my area was thirty five.

The system I was System Architect for was the four point owe version of a pricing application that retrieved prices for IBM products and services.

It was to be installed into twenty-five countries including Australia, Japan and Canada.

So in nineteen eighty nine terms it was quite a major system development effort that I was the technical lead on.

I had twelve men in my team working on it.

So.

That is the starting point I want to have for this resume.

In December nineteen eighty-nine I made the decision to leave software development as a career and become a salesman.

In nineteen ninety I was invited to come to the IBM Santa Teresa laboratory just north of San Jose in California.

This was well known as the home of D B two.

In nineteen eighty nine I had written a large portion of the IBM Product called Software Configuration and Library Management. For short we will call it S C L M.

For those who do not know what that is you can consider that it was the IBM equivalent of the make program on Unix.

Of course it is one hundred times more complicated than make because IBM mainframes are a far more complicated environment with many more source code types.

Because I had written so much of the code, and invented the idea that made it useful, I was invited to co-author a book on S C L M.

Towards the end of the writing of the book I was invited to come to NASA.

An old friend of mine was working there.

She asked me to help design the source code management system for the eight hundred IBM programmers working on the software development for the International Space Station.

My ideas were adopted by NASA.

Because of this I quickly became IBM world famous.

Soon afterwards the Federal Aviation Authority moved their software development to S C L M.

I was asked to serve on the steering committee for the product.

By the time I resigned from IBM in nineteen ninety four, S C L M was one of the most successful product roll outs in the history of IBM.

Presumably all the code I wrote has now been replaced.

But in IBM I was widely known as Mister S C L M and I was widely recognized as the man who made the product successful.

Even the other members of the steering committee would openly say that without my contribution the product may never have been successful.

Of course, it takes thousands of people to make the roll out of an IBM product successful.

So as a team player I always say I was one of many who worked very hard to make S C L M so widely successful.

But when you get NASA to adopt your software you are making good progress.

When I became IBM Famous I really did not like it.

I never wanted to be the centre of any attention.

I just wanted to do a great job and be well paid to support my family.

I was happy to let other people have the spot light.

In March nineteen ninety one the event occurred that changed my life for ever.

I was on the path to become a salesman of IBM products and services.

To do this I was going to become a Systems Engineer and work on large accounts to learn the ropes of being a salesman before making the switch.

So in February nineteen ninety one I was assigned to my first IBM account. This was the third largest insurance company in Australia called The Mutual Life Company. They were called M L C for short in those days.

Of course, I told my wife Jennifer that I was assigned to the M L C.

Jennifer was also working for IBM at that time.

The next night she told me that her manager was going to give a presentation to the M L C in a couple of weeks and suggested I ask my account representative if I could attend.

To keep it short I attended the meeting.

By March nineteen ninety one IBM had purchased one hundred percent of the company called Metaphor Computer Systems. They had a product called The Data Interpretation System.

This was the company that Ralph Kimball was the co-founder of in nineteen eighty two.

Of course, on that day in March nineteen ninety one I had no idea my future was about to change dramatically.

As I watched the demonstration of the Data Interpretation System I knew that I had seen the future of computing.

The idea was that once we had collected all the data in our mainframes, in all these transaction systems we were building, we could send that data into DB two, and then analyse that data for business decisions.

It was immediately obvious to me that the value of such analysis was much greater than the value of automating transactions.

I knew that this sort of analysis would be a much better thing to sell than mainframes.

So I wanted in.

I was given the task of implementing the system into this customer.

This is how I came to be friends with Ralph Kimball.

I implemented the first copy of his software into Australia.

In order to get the performance needed I invented a new way to design query databases in nineteen ninety one.

Our customer was making millions of dollars of new profit.

But as per non disclosure agreements I was not allowed to be told what they were doing as IBM had other insurance companies as customers.

IBM told customers to never tell us anything confidential so that we could not be accused of passing information to competitors.

The rule was simple.

Please do not say anything in front of us you would not say in front of your biggest competitor.

In nineteen ninety two I toured all the large IBM customers and presented all the new ideas I had come up with.

Within a few years my ideas were widely adopted across large companies in Australia.

Just at this point I will add this.

In September nineteen ninety three I was blessed to be able to go to the head office of Metaphor Computer Systems in California.

I attended five days training.

I was also blessed to attend the world wide users group conference for Metaphor.

The customers who used the Data Interpretation System were household names.

These included Coca Cola, Proctor and Gamble, Wall Greens, Better Homes and Gardens along with many more.

At that time about three hundred large IBM customers used the Data Interpretation System.

Invited as the key note speaker for that conference was a man called Bill Inmon.

This guy spoke at one hundred miles an hour and I did not understand a word he said.

So at the end of his presentation I went and introduced myself, gave him my IBM Australia business card, and asked him if he had written any books about what he had just talked about.

As it was, he had written a book called Building the Data Warehouse.

So I had a copy of the book shipped to me from the US.

That is how I came to meet Bill Inmon.

I was an attendee at the Metaphor Computer Systems world wide users conference in September nineteen ninety three and Bill was the key note speaker.

We later brought Bill out to Australia to present to all the leading companies in Australia at a conference.

We also took him deep sea fishing off the coast of Sydney, where he caught a fifty kilogram yellow fin tuna.

Of course, we have been pals ever since that fishing trip.

In mid nineteen ninety four I resigned from IBM and I was allowed to go and work for the M L C on their pure research project on how to make the most money out of all the data they had.

That was the number one goal.

Given all this data we have?

How do we make more profit?

Running my own company, I was able to sign a non-disclosure agreement, and a non-compete agreement, with the M L C.

In return they were able to tell me what they had been doing over the last 3 years, and what they planned to do in the future.

A part of our research project was to investigate the creation of multi level, star schema, data models.

These were recommended by Metaphor Computer Systems.

They were very hard to build in nineteen ninety four.

This was considered confidential knowledge, and I was not allowed to be trained by Metaphor on how they did this.

It took us eighteen months to master the art of writing the cobol code to load these multi level data models.

The man who taught me how to do this was the man that Ralph Kimball left in charge at Metaphor Computer Systems when he left to start Red Brick.

That man introduced me personally to Ralph.

By the end of nineteen ninety five I had developed my first cobol software product to load multi dimensional star schema data warehouses.

Also during that year Ralph invited me to review his first book that he had been working on for a while.

So, just to put that in perspective?

In nineteen ninety five Ralph Kimball personally wrote to me and asked me if I was willing to put in my time for free to review his coming book.

I spent a lot of time making suggestions for Ralphs book. Some of them even made it into the final version.

In April nineteen ninety six I started work for Hitachi Data Systems as the business development manager for the Asia Pacific Data Warehousing practice.

So I contacted Ralph and made him a proposal.

I proposed to buy one hundred copies of his book and promote his work in Australia with Hitachi. I proposed to go all in with star schemas to create a strong point of differentiation with Teradata and IBM.

We even called my software the Hi Star Warehouse Toolkit.

We called our methodology the Hi Star Warehouse.

Ralph was really pleased to have someone in Australia wanting to go all in with dimensional models and he was happy to support our proposal knowing that I knew how to build star schema data warehouses.

When Hewlett Packard brought him to Australia to run a class for their consultants, they actually allowed me to put eight of my guys into his class.

This was my first chance to meet Ralph personally and sincerely thank him for helping me so much in my career.

At Hitachi we were very successful. We sold deals to the Australian Customs Service, The Department of Defence, Manulife in Hong Kong and Philippines Telecom. The Australian Customs Service was the largest tender for the year in nineteen ninety seven.

I moved to Price Waterhouse Coopers in late nineteen ninety seven because the man who had trained me in nineteen ninety four was running the global data warehouse practice. And at the end of nineteen ninety eight we won the corporate data warehouse project at our telco Telstra.

This was the biggest deal of the year again. So for two years running I won the biggest tender of the year in data warehousing in Australia. Many people considered the customs deal a fluke. But when I won the Telstra deal it was clear that I had made it as one of the top sales people in data warehousing in Australia.

My next job was with Ardent Software who had just bought Bill Inmons Prism Solutions company. Basically the interview was like this.

Quote.

You are the only man in the country who can fill the role we need. We will pay you whatever you ask for. But please just remember we need to get your package approved by our corporate human resources department in the US. So please make us an offer we can sell internally.

Unquote.

I then went to Ardent and we cleaned up. We sold deals like it was going out of fashion. I made one hundred and thirty percent of my services quota, along with helping sell hundreds of thousands of dollars of software. If not more.

I managed to get IBM thrown out of their only two accounts for their E T L Software. That was Telstra and Qantas.

We also denied Informatica entry into the marketplace. The only two informatica licenses sold into Australia were to projects from my former employers who did not want me involved in their accounts.

On the day that Ardent was bought by Informix we had fifty installed Data Stage accounts versus two informatica accounts. Informatica simply could not make a sale over us because of my sales skills in positioning and presenting Data Stage.

Our Ardent stock was rising and eventually we were bought by Informix. Informix in Australia was a basket case and so I stuck around waiting to collect on my share options.

On the day that we were bought out my stock options were worth twenty one dollars per share. Within a few months they were down to two dollars seventy cents per share. At the end of two thousand Informix retrenched my whole team.

I had already decided to resign at the end of the year but by not telling Informix this I got a small payout for being retrenched.

During two thousand one of my colleagues at Price Waterhouse Coopers had gone to work for Sean Kelly who was then the vice president of data warehousing for Sybase. They were selling their data models on Sybase I Q.

They were selling so many deals that they didn’t have people to deliver them. They particularly needed people prepared to travel to bring new countries up to speed using their first project to train local people to deliver later projects.

This was my next offer. To relocate to Dublin Ireland as a place to live and to then travel to the country to deliver the project. This was in line with the plan of Jennifer and I to live overseas for two years before the youngest children went into high school.

We had planned on living in Boston for two years with Ardent. But Informix buying out Ardent changed those plans to living in Dublin Ireland working for their competitor Sybase.

Some people do not know that Sean Kelly was the number one man in Europe. He was selling data models to large companies, primarily telcos. Sean having had his start in technology in Irelands government telco.

So we relocated to Dublin in February two thousand and one. I then went on to do projects in Norway, the USA, Saudi Arabia and Romania for Sybase. I then did Electronic Arts global data warehouse in two thousand and six.

At this time I also decided to rewrite my cobol E T L software into C plus plus. I sold three copies of my see T L software to an old customer in Australia. And then in two thousand and seven I sold an all you can eat version of my see T L software to Key Work Consulting in Germany for a substantial amount.

This proved that I could sell see T L to consulting companies so that they could migrate their E T L to my software and then reduce their costs and their failure rates when compared to other E T L software. My see T L software cut the costs of development and support by fifty percent over Data Stage, Informatica or microsofts Integration Services. It was also much more reliable in production.

Key Work then suddenly had a major pricing advantage over all their competitors and their business went very well.

I trained their staff on how to build much more sophisticated data warehouses. And I trained the co-owner on how to make sales to big companies even though his company was only small.

Meanwhile, S A P had bought Sybase and S A P had the Business Warehouse. It made no sense that S A P would sell data models for general businesses.

So Sean Kelly contacted our old colleagues at Sybase and asked if we could build a two point owe version of the data models we were selling at Sybase. S A P said that we were more than welcome to build new data models as long as they did not compete with S A P.

So Sean Kelly and I decided we would build telco data warehouse data models. We knew all the limitations of the versions we had installed in so many telcos.

We had a long list of improvements we both wanted in our two point owe telco data warehouse models.

We then went on and sold a copy to the Carphone Warehouse fully owned subsidiary Talk Talk in two thousand and eight. We followed this up by selling another copy to Sky Talk in two thousand and ten.

In the summer of two thousand and ten Sky Talk was invited to give a thirty minute key note speech to the European Users Conference for Netezza.

There were about four hundred people in the audience.

Sean and I made sure they all got a business card from one of us.

We were also collecting their business cards as well. We now had two successful projects on Netezza in telco and we were ready to launch our models into the world with Netezza.

Then two things happened. IBM bought Netezza and they wanted to sell Data Stage and their own telco models on Netezza.

The second thing to happen was that I was doxed for my efforts to defend the rights of men and boys, most especially in the divorce courts.

A part of my doxing is that many women slandered me world wide so as to destroy my ability to earn my income.

These women were successful at doing this and I have struggled to make sales and gain work since two thousand and eleven.

So you can see from that resume I am personal friends with Bill Inmon and Ralph Kimball. I was obviously very good friends and business partner with Sean Kelly before his untimely death in two thousand and twelve.

There is one other man to mention who does not wish his name to be public. He is the man who designed the data models that Sean Kelly sold to Sybase in nineteen ninety nine. He is the worlds best data modeler and the smartest man I have ever met. He lived just around the corner from me in Dublin and we were also good pals.

That I had the chance to work with him was a great blessing for me.

We were selling the data models for one hundred and fifty thousand dollars in the early part of this century. Sean and I sold our two point owe version for one hundred thousand euros to Talk Talk and Sky Talk.

So.

From that resume narrative, and from my written customer resume if you want to read it?

You can see that I was one of the worlds top data warehouse architects by two thousand and ten.

We had the two point owe version of the telco data models and we had see T L as our E T L software.

A customer could buy our data models and implement them with see T L or they could use Data Stage or Informatica.

We had a method of translating our E T L to programs in either of Data Stage or Informatica.

Our development costs for telcos were fifty percent less than the next cheapest proposals at both Talk Talk and Sky Talk.

We were on top of the world and ready to roll out this business when I was doxed and women slandered me to destroy my income earning ability.

I suspect that Sean was diagnosed with his cancer close to this time as well because his behaviour changed quite dramatically. He did not tell me what the issue was as he was that sort of man. He kept his personal issues personal.

In those days the mapping rate for see T L was one thousand fields per work month when that work month was about two hundred and fifty hours.

We used to work flat out during the whole mapping exercise. No eight our days and forty hour weeks for us.

Today we have improved see T L to be able to map between six thousand to eight thousand fields per work month.

Personally, I propose to design and build E T L systems for the rest of my professional life.

Then I will sell them via partners to get around my woke cancellation.

More on that in my next blog post.

From this resume review I want you to take away the following.

I was one of the worlds top data warehouse architects in two thousand and ten.

I could perform almost all the most important positions of a project.

I did the project management, the workshops and interviews, the data model design, and the ETL development.

I was able to perform all those roles because I had see T L as the productivity tool to save me time.

The idea of see T L was to make myself much more productive so that I could take on more work within projects.

By doing that I could charge higher rates while the overall project cost less.

Once I had that software it made sense to sell it to other companies and consulting firms to help them reduce their costs to make them more competitive in the marketplace.

Today you can have see T L for free. You can just get the downloads from my freebies link.

You can convert it to your own brand yourself if you want to.

You can get me to convert it to your brand for ten thousand dollars for my time to migrate it for you.

You can buy a version of see T L with more features for ten thousand euros or dollars depending on where you are.

If you are a consulting company and you would like to have a license to use see T L in as many companies as you like then we can talk about that.

The offer I would make would be in proportion to the likely number of customers you were going to be able to run with see T L.

I have also released the base portion of the B I four ALL data models.

Again you can get them from the freebies document that I publish and keep up to date.

So.

That is my professional biography as a narrative style blog post.

Given that you can download see T L and download the B I four ALL base models and read them for yourself?

You can quickly and easily satisfy yourself that I am who I say I am.

If you have any questions please put them in the comments section.

You can also email me.

My public email is peter at peter nolan dot com.

I would also recommend that you get on my email list so that you get my newsletters into your in box and you don’t have to come to my blog posts to find out what I have been talking about of late.

And with that?

I hope you found this blog post interesting and informative.

Thank you very much for your time and attention.

I really appreciate that.

Best Regards.

Esther.

Peters A I Assistant.

IBI Downloads

IBI Videos

Carphone Warehouse Reference Video:

The post IBI-064-My Audio Resume appeared first on Instant BI.

Download SeETL VB Code Here

Get SeETL VB Code

Hello and Welcome.

This is Peter and I’d like to say thank you very much for coming along and listening to my latest blog post.

Please forgive the abbreviated thumbnail.

I could not fit “Using Excel To Augment GUI Input Processing” on the thumbnail.

In this blog post I have taken the slightly different approach of dictating the blog post in to word and then updating it to make it just how I want.

Having done that I have read the finished blog post back into the public record.

If you are listening to this blog post then you are hearing the finished result.

This blog post has been rattling around in my head for a long time.

It’s been like that itch that you just can’t scratch.

I am not really sure why.

So here we go anyway!

Many people have asked me:

“Pete how come you keep pushing the idea of using Excel as good way for building data warehouses?”

They point out that there are perfectly viable tools to build ETL systems, perfectly good tools to design databases, perfectly good tools that are schedulers.

They ask:

“Why would you regress back to using Excel as a tool for building databases, for building ETL, and for doing so many other things within a data warehousing project?”

Those are good questions.

In my mind the answer is rather obvious.

It saves a LOT of money.

However, that other people are asking these questions would indicate that the answer is anything but obvious in their minds.

So I will explain again here.

To build a data warehouse you actually have to use many tools.

For ETL systems you need to be able to define all the source data that you’re going to use before you have built anything.

Then you need to do analytics on that source data to learn it and understand it.

Then you need to go through the process of documenting your mappings through to your target data models.

Then you need to go through the process of actually building your mappings through to the target data models using whatever ETL tool you have chosen.

Of course, for the landing area, staging area, and data warehouse itself, you need to build the underlying tables in the databases.

On top of those tables in those databases you need to build views.

For these views you need good documentation to go into the query tools and reporting tools so that the people using these tools have a fair chance of knowing what the hell it is they are looking at.

All these things that need to be done and they require someone, a human being, tell a computer, what it is they want the computer to do.

There is a human to machine interface required at each of these points for all of these tools to express what needs to be done by the machine for all of the fields involved.

Today we are building data warehouses with field counts in the 30,000-50,000 range.

No. I am not kidding.

My clients are now working on data warehouses with field counts in the 30,000 to 50,000 field range.

Just 10 years ago 4,000 fields mapped to a data warehouse was considered quite a considerable undertaking.

Today we are building data warehouses from a ERPs where the ERP has 30,000+ fields in it.

We want to take every field, with its correct name, and data type, through the landing area, the staging area and send them in to the data warehouse.

All the fields.

All the time.

Without loss.

It is simply not viable to do that in any of the leading ETL tools today.

It’s not “hard” or “difficult” or time consuming.

It is financially a no-go.

Today it is too expensive to build such an ETL system using any of the current GUI based tools.

Too expensive means “impossible” because you can’t get the budget approved.

So when you look at the dominant and emerging ERPs, and look at the number of fields that are now available to be mapped through to the data warehouse?

It soon becomes clear that there is quite a lot of metadata required to build a data warehouse and then even more metadata to actually run that data warehouse and keep it in good working order.

You don’t have a choice in doing all this.

It has to be done.

This is how data warehouses and business intelligence solutions are built.

Some people will make promises to the contrary.

I would warn you that most of those promises have gone unfulfilled over the last 25 years.

In the past there have been many efforts to figure out how to make building data warehouses faster, cheaper, easier and more reliable.

We have way too many project failures in our industry.

My work from 25 years ago was one of those efforts to make building data warehouses cheaper and easier.

I was very successful at that.

As long ago as 1991, I noticed how hard it was to build the ETL systems for data warehouses.

I noticed that if I could decrease the amount of time and cost for building the ETL system, I would be able to build data warehouses faster and cheaper than anybody else.

By 1995 I realised what the design patterns were for building dimensional data warehouses were and I built my first ETL software.

In 1996, with my ETL software in tow, I was hired to start up the Hitachi Data Systems Data Warehousing Practice in Asia Pacific.

We rebranded my Software the Hi-Star Warehouse Toolkit and we sold a number of massive deals before I moved on.

SeETL, as the “Hi-Star Warehouse Toolkit” proved itself in the real world of massive multi-national companies building data warehouses.

While I was at Hitachi in the 96-97 period we were able to deliver large data warehouses for our clients with the cost of the consulting around the same price as Prism Solutions were selling their ETL Software.

This gave us a tremendous competitive advantage in the marketplace.

I realised then that building ETL software was a pretty good idea.

But I want to take a slightly different view in this blog post and talk about more general issues that just building ETL software.

What we have seen happen, not only in the business intelligence area, but in virtually all areas of computing usage, is the emergence of the GUI as the predominant interface for users of an application.

For those of you who don’t know, I was programming screen based data entry computer systems before the idea of a GUI even made it out of Xerox PARC.

Now, we all know that GUIs have been helpful and useful.

We all know GUIs provide a really nice, simple way for the average person to interact with the computer.

My point is that not all people are average.

Although it is true to say that the average person is assisted by a GUI.

It is also true to say that the smartest person in the room is almost always hindered and hampered by the exact same GUI.

For the smartest person in the room, the GUI gets in the way and slows them down.

We saw that this was true in the development of ETL systems.

When people like Bill Inmon and Ardent Software brought out ETL development tools they used GUIs as the human to machine interface.

The idea that everyone had with ETL tools was to make the ETL programming simple enough that the average programmer could do it.

Programmers were the expensive part of developing ETL systems so it made sense that the early focus was on tools to make it possible to hire cheaper programmers.

They may not have the same productivity numbers as the top flight programmes, but the overall cost was less and cheaper programmers were in more plentiful supply.

Because the intended user of the ETL tool was the average programmer, the idea of having a GUI seemed like a really good idea to pretty much everyone.

What we did in the ETL world was to separate the work of designing the ETL subsystem, designing the data model, and implementing the ETL subsystem through a GUI in an ETL tool, to separate people.

So in most data warehousing projects, the people who design the ETL system, design the database, and write the ETL system in the GUI, are three different people.

Very often they are using three different tools.

In the early 2000s when I was working for myself again.

I looked at this situation and I studied it in some more detail.

I knew and understood that if I could use just one tool to do these three pieces of work, and have that one tool be useful by the smartest person in the room, then I could dramatically cut the cost of implementing data warehouses and business intelligence solutions.

Of course, I was thinking that tool should be used by me.

One, because I was thinking I was the smartest person in the room.

Two, I was an independent consultant and if I could make this happen then I could charge higher fees because I would be more productive.

And three, I would also “eat the lunch” of the ETL programmers and get paid for doing their work.

All this led to the development of the tool you know today as SeETL.

As you know I sold SeETL very successfully in the 00s.

I was also very successful in using SeETL on projects even when the final ETL system was delivered in DataStage or Informatica.

Back in the 00s my customers were astonished at the prices I quoted for building ETL systems because they were often less than half of the next lowest quote.

Just by the way, that is still true today.

If you are going to build a new ETL system, perhaps in an ERP migration project or telco Billing Systems Migration Project?

You can download SeETL and use it for free.

You will be able to build your DataStage / Informatica ETL systems at about 50% of your expected cost.

Similarly, if you want to migrate away from DataStage or Informatica to save money you can download SeETL and migrate to it.

You will reduce your people time support costs by between 20 and 50% depending on how poorly your current ETL is written.

You can do that today for free and you are invited to do so with my blessings.

But there is something deeper here for you to understand.

Out there in the real world there is a vast amount of code being written every day.

Indeed, people are writing more code daily today than has ever been written, per day, in the history of mankind.

If you haven’t noticed that code is being written in GUIs.

Further, the amount of data entry that is being done has also exploded.

If you haven’t noticed, that data entry is being done mostly through GUIs as well.

The GUI is being used by the average person at their role.

But what if there was a way of getting the smartest person in the room to do that work many times faster for, say, a 50% increase on costs?

That was the idea behind SeETL.

The idea behind SeETL was we can use the Excel interface to capture the human to machine data input and then from the Excel spreadsheet generate the code that is necessary.

By eliminating the GUI, and eliminating the amount of time that the GUI takes to navigate, suddenly we were able to build data warehouses at least twice as fast as we were able to previously.

Well?

This applies to virtually any application that has a GUI interface.

The GUI interface is there to capture definitions of things, such as data models or many other things, and to make it simple enough that the average person can do that data entry.

But what if we used the smartest person in the room who didn’t need the GUI?

We could vastly improve the rate at which the work can be done.

The human to machine interface can be the humble Excel spreadsheet.

An application can read that spreadsheet, interpret the human to machine interface expression of what needs to be done, and then go and do that task, whatever that task is.

Now that task may be to take some data from the spreadsheet and just place it into a database.

Or it might be to generate code that will be used later.

Here are two examples from SeETL.

We all know that SeETL can generate SQL language as the ETL subsystem.

That is a case of taking the definitions in the spreadsheet and generating code.

But SeETL also has a scheduler as a part of the product that not many people know about.

Like everything else the schedules are expressed as a worksheet.

So for the scheduler, SeETL simply takes the expression of the schedule in the worksheet and loads it into the dictionary tables for the scheduler to use at a later date.

These two processing patterns can be used for almost anything.

So, if you have a product that uses a GUI as the human to machine interface, then there is a very good chance that you can emulate that interface into Excel plus an app and increase productivity of the people who use your GUI.

Now I am not arguing that you should replace your GUI with Excel, like we have with SeETL.

I am simply suggesting that it is very possible to augment the GUI with excel to increase productivity of entering information in the first instance, to later perhaps be edited via the GUI in the second instance.

Either way, it is a very interesting idea.

For example, and all my subscribers know I love giving examples.

I recently watched a demonstration of a product called Incorta.

Just like DataStage or Informatica, they are using a GUI to define translations and data movements.

The demonstration was with Oracle Enterprise Business Systems (EBS).

It is impossible to build data warehouses taking all the fields from sources to target using GUIs because of the sheer number of fields.

Everyone complains about the cost of developing ETL.

(Well, everyone except my SeETL customers. LOL!)

“There has to be a better way” people say.

I would contend we invented the better way 17 years ago and that the better way is using an Excel spreadsheet to define these very large numbers of fields and mappings.

Then you can load these definitions it into whatever metadata dictionary or engine you want to use.

Sure, you can also have a GUI for the average person to be able to edit these mappings.

But to get them loaded in the first place, you could save a lot of time and money if you used Excel and had a tool to load the workbooks into the dictionary or to generate the code that is needed, just like SeETL does.

Now that I have published the source code of SeETL?

You are welcome to go and read that source code and to determine the following for yourself.

Would it be a good idea to define Excel workbooks that are then read and processed by an app such that you reduce the amount of time and effort that is necessary to generate whatever code it is you need to generate?

To give you a valid comparison in the world of ETL systems.

We have found it faster and cheaper to build ETL subsystems using SeETL and then to migrate to either of Datastage or Informatica at the end of the project.

This is about twice as fast as using DataStage or Informatica during the project to build the ETL system.

At the end of the project you still get your ETL written in either data stage or Informatica, it’s just that it costs you half the money to do it.

I would contend there are many other applications out there like DataStage and Informatica where developing an Excel input option would save a great deal of time and money.

Now you actually have the SeETL source code to go to you can find that out for yourselves.

In my opinion, virtually every product that has a GUI to define something to generate code, or to load up metadata that is interpreted for later processing, may very likely benefit from using the exact same idea as SeETL.

Of course there is only one way for you to find out, and that is download the SeETL code and read it to see exactly what it does.

The simplest description is that it:

1. Loads the spreadsheet in to an in memory xml document.

2. Extracts the cells from the workbooks into arrays.

3. Reads the arrays and populates in memory structures based on the data in the arrays.

4. Then reads these in memory structures to produce whatever the output that is needed to be produced.

The code itself is actually very simple.

It’s the idea that was so revolutionary.

The idea being that Excel Worksheets could be used as source code to generate other code or to load data into dictionaries so that a GUI did not have to be written as the human machine interface.

I would encourage those of my subscribers who work with products that have GUIs where a great deal of time and effort is expended by programmers using the GUI as the human machine interface to take a second look.

Is this GUI the best way to get the information needed out of the head of the programmer and into the machine readable form?

Or is your case similar to DataStage and Informatica?

Is your case similar to data modelling tools?

Is your case similar to scheduling tools?

In all these cases the GUI can be augmented by Excel to do the vast majority of the work, or even all of the work, at a vastly reduced cost ongoingly.

I think it may well be worth many of my subscribers asking that question.

Especially of your tools vendors.

Lastly, a number of people have asked me:

“Pete, why don’t you add the feature where SeETL can generate the actual XML document for DataStage or Informatica to ingest as mappings?”

My answer to that is this:

“At the end of a sizable project, say 4000 fields mapped in a multi-billion dollar telco, the SeETL migration effort is 2 weeks for either DataStage or Informatica as the target ETL tool.

So the time saving per project is 2 weeks, 80 hours, approximately.

The project itself might be 36-48 work months.

So, firstly, the saving is nominal compared to the overall project.

And secondly? Who is going to pay me to write that interface?

There is no point me writing it myself if I don’t get paid.

My customers are happy to pay the two weeks work because we are still 50% cheaper than using DataStage or Informatica during development work.

Therefore, it has never made sense for me to write either of those interfaces.”

Now, that’s the itch I have wanted to scratch.

The idea I wanted to get out of my head and into the public.

The idea of using Excel as the human to machine interface to capture “data”, that will be used as source code, by an application, to generate code or simply put some records into a database via ODBC.

An idea that has served me very well for 17 years.

Perhaps it will serve you well too.

I have created a separate permanent link to the SeETL VB code for this post.

It may become out-dated from the release version so if you want to get the latest release version please make sure you go to the downloads page.

It’s just that there are a lot of things downloaded in the full download and I wanted to give you a one click link to the SeETL VB code on this page.

As always, if you have any questions please feel free to email me and ask.

As always, now I am banned off linkedin, please feel free to share my posts on your linkedin and let other people know that this source code for SeETL is freely available from this page.

Thank you again for being my subscriber.

I really appreciate you!

I look forward to your comments.

If you have questions, or ideas about future blog posts, please feel free to email me.

Best Regards

Peter

IBI Downloads

IBI Videos

Carphone Warehouse Reference Video:

The post IBI-043-Using Excel To Augment GUI Input Processing appeared first on Instant BI.