Big Data, as the name suggests, is about data that is BIG in nature. The data is BIG in terms of size, and it is difficult to manage such enormous data with relational database management systems that are quite popular these days. This introduction to Big Data starts with the basics.
Big Data is not just about being large in size, it is also about the variety of the data that differs in form or type. Some examples of Big Data are given below :
- Scientific data related to weather and atmosphere, Genetics etc
- Data collected by various medical procedures, such as Radiology, CT scan, MRI etc
- Data related to Global Positioning System
- Pictures and Videos
- Radio Frequency Data
- Data that may vary very rapidly like stock exchange information
Apart from difficulties in managing and storing such data, it is difficult to query, analyze and visualize it.
The characteristics of Big Data can be defined by four Vs:
- Volume: It simply means a large volume of data that may span Petabyte, Exabyte and so on. However, it also depends organization to organization that what volume of data they consider as Big Data.
- Variety: As discussed above, Big Data is not limited to relational information or structured Data. It can also include unstructured data like pictures, videos, text, audio etc.
- Velocity: Velocity means the speed by which data changes. The higher is the velocity, the more efficient should be the system to capture and analyze the data. Missing any important point may lead to wrong analysis or may even result in loss.
- Veracity: It has been recently added as the fourth V, and generally means truthfulness or adherence to the truth. In terms of Big Data, it is more of a challenge than a characteristic. It is difficult to ascertain the truth out of the enormous amount of data and the one that has high velocity. There are always chances of having un-precise and uncertain data. It is a challenging task to clean such data before it is analyzed.
Big Data can be considered as the next big thing in the IT sector in terms of innovation and development. If appropriate technologies are developed to analyze and use the information, it can be the driving force for almost all industrial segments. These include Retail, Manufacturing, Service, Finance, Healthcare etc. This will help them to automate business decisions, increase productivity, and innovate and develop new products.Thanks Jasjeet Singh for an excellent write up.
After This Introduction to Big Data: Where It Meets SQL Server
You do not have to leave SQL Server to work with large and varied data. Columnstore indexes compress big tables and make large scans and aggregations much faster, which covers the volume side for many warehouses. PolyBase, added in SQL Server 2016, lets you query data outside the database, such as files in cloud storage, with plain T-SQL through external tables. For variety, SQL Server can store and query JSON and XML next to your relational rows.
My advice is to start with the problem, not the label. If your data fits on one well-tuned server, a relational database is still the simplest answer. Reach for distributed tools only when the volume, speed or shape of the data really needs them.
Published by Pinal Dave on SQLAuthority. More of my work at pinaldave.com.
5 Comments. Leave new
Great article. But how are scientific data stored now, do they follow relational database model and having trouble, which leads to the discussion on ‘Big Data’ ?
you have make ‘Big Data’ to small & sweet blog post,
can you write same about ‘NoSQL’ in short & sweet
I liked the part where you listed the examples of Big Data. Thanks for the article..
The 4 V’s come from IBM’s website: … it should be acknowledged
How Microsoft is planning to accomodate big data? Are we going to have some different kind of SSMS or a next gen of query engine is gonna be launched? How do we prepare ourselves to cope with Big Data on Microsoft platform?