TARGET AUDIENCE : Professionals with the background of MIS, WEB DB, ETL technologies like VB, Delphi, Perl , Java, Dot net , Oracle , Teradata, Sap HANA , Vertica etc .
This is an old write up , was missed earlier. please refer BD as BIG DATA.
Big data project initiation at C level, being discussed and debated that
- How big-data solutions can help their organization?
- Is it worth sparing budget on big-data solutions?
- Will the returns on investments be worth?
At the same time IT delivery lines of vendors are increasing competency to deliver solutions. Small attempt to see this technology from use case perspective.
On the other hand we CODERS have some curiosity that what is big data and how it is different form current data……even some feels that after new BD projects data warehouse projects will no longer be required. Scary for CODERS ,,Though i am not of this opinion. B data can not eliminate data warehouse needs.
Is big-data so big that it cannot be managed by current RDBMS systems? Question is rational, for a person who plays with RDBMS and DATA WAREHOUSE in a day to day life.
End result of BD processing is also data analytics. Does that mean data warehouse systems will no more be useful and ETL tools will not be required as we have Hadoop and other BIG DATA tools in place?
Here are some good discussion points on big data….
1) BD analytic will eliminate the need of Data warehouse.
There is a basic difference in BD and data warehouse systems…… BD can take care of data with velocity but in data warehouse we get the data with latency.
Current data warehouse system will have their place and cannot be replaced by big data. There is a probability that way we fill data warehouse now gets changed over period of time …
2) Since BD need higher processing capabilities, ETL tools will no longer be required as Hadoop will do it.
ETL will co-exist with BD…… Same way ETL tools will also exist as there will always be need to data consolidation from different sources and data loading.
Hadoop can manage processing (mining) activity but ETL will have its own place to do extract and load.
ETL landscape might change a bit and there are probabilities that new players come in with the benefit of lower acquisition cost of extract and load capability
3) BD analytic is a huge social media data analytic s?
Not Always , BD analytic is more then social media analytic s.
4) RDBMS systems will not be capable of handling BD.
I am not sure as of now that RDBMS can handle , but there is a probability that newer version of RDBMS can handle it , but in memory DB like Vertica , SAP HANA, Mongo DB can easily handles BD.
1) Definition – what is big data?
It is not the size which defines that data is BData. The key is type and frequency. BD is generally referred as both unstructured and structured data which continuously comes in your systems and fills up all your data buckets………
Data processing capability for huge amount of unstructured data is BData analytic. BD doesn’t only means social media data analysis and reporting there are organizations which are processing terabytes of internal data and BD processing is not new for them.
Some real world use cases to understand the need of big-data projects
- System Scenario 1 : Live (real time not real world) fleet health management and analytic.
- System Scenario 2 :Sustainable and efficient traffic management.
- System Scenario 3: Wildfire, Real time detection and reporting.
- System Scenario 4: ANY analysis.
- System Scenario 5 : efficient weather reporting.
- System Scenario 6 : Plant operations optimization.
Above are just sample scenarios, there are around 2000 use cases I could see by myself.
Hope it can give a new or different perspective on big-data ? any thoughts ?