Initially i was a confused and reluctant ETL tool user. You know why ? Because, i was frustrated from my Java, Dot net, friends who wanted me to work in their technologies.
It was just because they wanted me to be in right profession and they used to think that ETL is not a go forward technology. We used to have long discussions( Believe me all used to lead towards not so good outcome ).
let me put up few of the perceptions generally non ETL people has about ETL Tools.
- ETL tools are expensive ( 😉 My take used to be that i like expensive things ).
- ETL tools only transform the data ( 😉 My take was, let us first do the cost benefit analysis, because no one will buy expensive produce just because it only transforms the data. Definitely there is something which we are not aware of ).
- ETL tools doesn’t provide any capability to enterprise apart from moving the data ( 😉 I was of the view, Are we really aware of bigger needs of enterprise IT ? or this is something which we needs to explore more ).
ETL Tools – What problem they solve ?
Now almost spending good amount of time with ETL tools i have built some understanding of solutions ETL Tools can provide. Here is a small attempt to share the same with you.
I will not talk difficult IT language which makes it difficult to understand for non IT people. ETL technologist doesn’t really need to understand that what is ETL tool and what value it provides ? because they already know it. Let me share few real time scenarios i have come across and actually they describe very easily ” The surprising effects of ETL Tools on enterprise IT “.
Scenario 1: Early IT Starters (Old School Enterprise )
One IT savvy enterprise who started in 1950’s. They were new and understood the benefit and started leveraging IT capabilities from beginning. They have used best technologies available at the time and built their systems around it. Technology evolved and they have upgraded their systems as per need keeping multiple cost factors in mind. Over the period of time they had multiple systems in multiple technologies.
Now for next Gen IT their all the systems needs to be integrated or in simple word they should be able to share the data across. So here is the deal, in multiple solutions one was ETL tools because ETL Tools makes it easy to pull transform and share the data across the applications.
You won’t believe that some of the organizations are having thousands of application serving their IT needs (3 000, 5000, 25000 also).
Do you want to know that why this kind of situation happened?, IT was business model of the organizations which lead to this situation. In Bigger organizations, individual business units were building application for their own needs instead of today’s single point IT delivery model. End result was above scenario. It is very apparent but Let me not discuss about the benefit of Tools here. Not good to expand the topic. Ok , I hope this real time scenario explains good about the need of ETL tools.
Scenario 2: We sale or Purchase company Type ( Mergers or Integration )
Merger or acquisition: Merger or acquisition leads to a position where IT application data need to be integrated or disintegrated. Do you know why? It is because; leadership need control over both the organization data, Isn’t simple.
Ok, let’s take one example to understand “Where one needs to integrate applications” Let us understand this with one real time industry scenario.
I am a manufacturer and produce boiler and all the parts associated with an industrial boiler. Somewhere down the line I have decided that I should purchase a company which makes best washer which can sustain the high temperature of steam while controlling the steam in tubes. Eventually I have purchased a washer manufacturing company as well. Being an organization now I want to track the washer company as well.
Here I will be very happy if someone can integrate both the organizations IT systems without much effort. (This is a realistic business need which cannot be ignored)
Now below explanation tells” what people look for in a solution which can integrate apps”.
One needs this to be done in fewer efforts. That means, Avoid building the new application or modifying the existing application. Moreover, one will be ok if someone can give me one view of few important reports which are strategically important.
At last but not least, no one wants to create a huge budget just for IT application integration.
Let us understand now “How ETL address above business needs”.
ETL Tools can give data to both the application as per their needs without many efforts. Because of this both the application need not to be changed and application code can be re-used. New code development can be avoided. Moreover it gives accuracy as you have not touched the application core algorithms.
This is also the Buzz word called “Lean integration”. Companies are trying to showcase Lean integration now a day.
Hum, ETL Tools can do this by sharing the data to parent company from new company.
Scenario 3: Data Analytic, EAI and DI
Ok, let me put this in simple way.
Broadly, or in real life also, we have no significance of data until and unless it is gets seen with respect to other data. Example!!! You have data about city traffic pattern, but do you think that this data will make any sense without relating it to location. After location it will make more sense if we can associate it with time. In same way, real world information technology (trend analytics) also relates different systems (Location, Population, City etc.) to see meaning full data.
With above statement we can conclude that different systems need to share the data to perform trend analytic.
Here ETL Tools comes in to picture. ETL Tools provide that capability with ease.
Enterprise data warehouse: ETL tools are capable of reading different data from different sources to integrate (relate) them. These upstream system works as a data feeding system. This data is later used by business to come to some conclusion on different aspects. This is called reporting systems in data warehouse. So we can say now that reporting systems provides different aspects of data which helps to start any initiative which is worth for business.
Scenario 4:Next Gen IT EAI
Forecasting application: You have a production unit and you want to forecast demand for the next quarter, so your vendor can supply the necessary stocks. Here ETL can pull data from different manufacturing units , stores, warehouses and after consolidation, it can give a rough picture about how much order needs to be placed for the next quarter.
Here was a small attempt to put forth business value ETL tools provide to an organization. Biggest fact of business is that it doesn’t run on technology complexity, capability of giving wider solution and the name behind it. Business IT needs cost effective development, enhancement, operations and easy scalability of solution.
Its a different matter that as an expert we are always interested in complex solution to keep us challenged. To understand this we need to understand the business needs and how ETL tools fulfill that need better than any other technology.
Any thoughts ?