# Chapter 4 — Data Science & Big Data ## 1. Data **Data** is a collection of facts, information, or observations about something. Data can be represented as: * Numbers * Measurements * Words/descriptions * Sound * Pictures * Videos --- ## 2. Data Analytics **Data analytics** is the process of examining and studying data to: * Identify patterns * Extract useful insights * Draw conclusions * Make data-driven decisions It may use mathematical calculations, statistics, charts, graphs, and other tools. **Example:** Recording temperatures throughout a day and creating a graph to determine how temperature changes. --- # 3. Data Science **Data Science** is an interdisciplinary field that combines **mathematics, statistics, data analytics, and machine learning** to analyze data and extract useful knowledge and insights. ### Data Science Pipeline **Data → Analysis → Patterns/Knowledge → Insights → Informed Decisions** It can be used to solve real-world problems. --- # 4. Key Concepts of Data Science ### 1. Data Collection of observations, facts, or information from different sources. It may be: * Structured * Unstructured Examples: numbers, words, audio, video, tweets, PDFs. ### 2. Dataset A **structured or processed collection of related data**, usually prepared for a particular purpose. **Example:** A collection of brain CT scans used to identify patterns in brain tumors. ### 3. Statistics The study and analysis of the characteristics and frequency of past data/events. ### 4. Probability The likelihood or possibility of future events occurring. **Memory:** Statistics → Past Probability → Future ### 5. Mathematics A fundamental part of data science used to: * Solve problems * Optimize models * Interpret complex data * Support decision-making ### 6. Machine Learning A branch of AI/computer science that uses **data and algorithms** to enable computers to learn from data and perform tasks. ### 7. Deep Learning A subset of Machine Learning that uses **artificial neural networks** to model aspects of human brain learning. ### 8. Data Mining The process of discovering **patterns and relationships** in existing datasets. ### 9. Big Data Very large and complex amounts of data that may be difficult for traditional data-processing systems to handle. ### 10. Predictive Analysis Uses **historical data** to predict future trends or events. ### 11. NLP **Natural Language Processing** is the study of interaction between **human language and computers**. Applications: * Chatbots * Language translation * Sentiment analysis --- # 5. Scope & Applications of Data Science Data Science is used to understand past trends, make decisions, and predict future outcomes. ### Industry * Data-driven decision-making * Quality control * Inventory optimization * Demand forecasting * Logistics ### Airlines & Transportation * Route optimization * Demand forecasting * Real-time tracking ### Stock Market * Predictive analysis * Quantitative analysis * Machine-learning-based analysis/trading ### E-Commerce * Recommendation systems * Customer segmentation * Shopping-cart analysis * Fraud detection --- # 6. Business Problems Solved Using Data Science Data Science can help determine: * Best route for shipping goods/passengers * Best product among alternatives * Possible delays in flights, ships, or trains * Suitable promotional offers * Popular products * Best delivery times * Ways to reduce costs * Future revenue forecasts ### General Process **Business Problem → Analyze Data → Find Patterns → Data-Driven Decision** --- # 7. Types of Data ## A. Qualitative / Categorical Data Describes objects or groups using **categories or labels**. Examples: * Colors * Cities * Grades * Transport types * Survey responses ### 1. Ordinal Data Categories have a **meaningful order/ranking**. Example: **A > B > C > D > E > F** **Memory:** Ordinal = **Order** ### 2. Nominal Data Categories have **no meaningful order**. Examples: * Car, bus, train * Red, blue, green * Lahore, Karachi, Islamabad **Memory:** Nominal = **Name/label** --- ## B. Quantitative / Numerical Data Data represented using numerical values that can be mathematically analyzed. Examples: * Height * Weight * Number of students * Growth rate ### 1. Discrete Data Values that are **separate and countable**. Example: Number of students = 20, 21, 22... **Memory:** Discrete = **Count it** ### 2. Continuous Data Values that can take **any value within a range** and can be measured to greater precision. Examples: * Length = 11.5 cm * Weight = 15.92 kg * Temperature = 23.47°C **Memory:** Continuous = **Measure it** ### Data Hierarchy **Data** → Qualitative/Categorical → Ordinal / Nominal **Data** → Quantitative/Numerical → Discrete / Continuous --- # 8. Sources of Data ## Primary Data Data collected **directly from the original source/respondent** for a specific purpose. Methods: * Surveys * Questionnaires * Interviews * Experiments * Direct observation * Sensors * IoT devices * Biometric devices **Memory:** Primary = **Personally collected** ### Respondent A person from whom information is collected during a survey or inquiry. --- ## Secondary Data Existing data that was **collected by someone else**, often for another purpose. Sources: * Government records/reports * Books * Journals * Research papers * Websites * Online databases * Previously conducted surveys/questionnaires **Memory:** Secondary = **Someone already collected it** --- # 9. Database A **database** is an organized collection of data stored electronically so that it can be easily **accessed, managed, and updated**. It can contain thousands or millions of records. --- # 10. Dataset vs Database ### Dataset A collection of related data prepared for a **specific purpose**, such as analysis or machine learning. ### Database An organized electronic system for **storing, accessing, managing, and updating data**. --- # 11. Database Management Database management involves: * Organizing data * Storing data * Accessing data * Updating data * Maintaining data --- # 12. DBMS **DBMS (Database Management System)** is software used to create, manage, and interact with databases. Examples: * MySQL * PostgreSQL * Oracle Database * Microsoft SQL Server --- # 13. Role of Databases in Data Science Databases are important because the amount of generated data is continuously increasing. They allow large amounts of data to be: * Stored * Organized * Accessed * Managed * Analyzed ### Supermarket Example A supermarket can use data to determine: * Which products are popular * Where products should be placed * How much stock is needed * Which products are commonly purchased together * Customer requirements This supports **data-driven decisions**. --- # 14. Data Storage After data is collected, it must be stored effectively. Storage methods include: * NoSQL databases * Data warehouses * Cloud-based storage * Blockchain --- # 15. Data Visualization **Data visualization** is the graphical representation of data to help examine it, understand it, and extract insights. Examples: * Charts * Graphs --- # 16. Summary Statistics Summary statistics provide a quick overview of a dataset. They may include: * Number of values * Minimum * Maximum * Mean * Standard deviation They help with: * Understanding data * Data cleaning * Data preprocessing * Feature selection * Visualization --- # 17. Big Data **Big Data** refers to very large and complex datasets that traditional data-processing software may struggle to manage. ## Three Vs of Big Data ### 1. Volume The **amount/quantity** of data. **Volume = How much?** ### 2. Velocity The **speed at which data is generated, received, or processed**. **Velocity = How fast?** Example: Real-time IoT data. ### 3. Variety The **different types and formats** of data. **Variety = How many types?** ### Types of Data in Big Data **Structured:** Organized in tables/relational form. **Unstructured:** Text, images, videos, etc. **Semi-structured:** Has some organization but is not stored in traditional tables. Examples: * JSON * XML ### Memory: **Volume = Amount** **Velocity = Speed** **Variety = Types** --- # 18. History of Big Data Big Data became especially prominent in the **early 2000s**. * Hadoop was developed to help manage and process huge amounts of data. * Growth of IoT increased data generation because more connected devices continuously produce data. --- # 19. Advantages / Uses of Big Data Big Data allows organizations to use huge amounts of information to: * Optimize processes * Make better decisions * Develop cost-effective solutions * Understand customers * Improve products and services --- # 20. Product Development Businesses can analyze data about customers and markets to develop products, services, and brands that better meet customer needs. --- # 21. Predictive Maintenance **Predictive maintenance** is a proactive maintenance strategy that analyzes existing data to predict when equipment, machinery, or products are likely to fail. This allows maintenance to be performed **before failure occurs**. --- # 22. Customer Experience & Satisfaction Big Data gives businesses a clearer understanding of customer experiences. Businesses can collect data from: * Social media * Website visits * Call logs * Customer interactions This helps improve customer satisfaction and services. --- # 23. Fraud & Compliance Big Data analytics can detect: * Unusual patterns * Suspicious activities * Anomalies This helps: * Detect fraud * Support compliance * Improve cybersecurity --- # 24. Challenges of Big Data ### 1. Data Quality Poor-quality data can cause: * Errors * Inefficiency * Misleading insights ### 2. Data Security & Privacy Protecting massive datasets from unauthorized access while maintaining privacy is difficult. ### 3. Rapid Growth of Data Data continues to grow rapidly, requiring systems capable of handling increasingly large amounts of data. ### 4. Big Data Tool Selection Choosing compatible tools that can interact smoothly with different platforms can be challenging. ### 5. Data Integration Combining data from different sources with different formats and structures can be difficult. ### Memory: **Quality → Security → Growth → Tools → Integration** --- # 25. Applications of Big Data Major applications include: **Healthcare → Media & Entertainment → IoT → Manufacturing → Government** ## Healthcare Big Data can be used to: * Analyze wearable-device and sensor data * Monitor patients in real time * Analyze electronic health records * Predict epidemic outbreaks * Provide real-time alerts * Predict serious medical conditions ## Media & Entertainment Big Data helps companies: * Understand audiences * Predict what audiences may want to watch * Target advertisements * Distribute content * Develop new business models * Monetize content ## IoT IoT devices continuously generate large amounts of data. Big Data analytics helps: * Analyze IoT data * Generate meaningful insights * Personalize customer experiences * Improve IoT capabilities **IoT → Generates Data → Big Data Analytics → Insights** ## Manufacturing Big Data helps manufacturers: * Use predictive maintenance * Improve product quality * Track production quality * Plan supplies * Predict output * Increase energy efficiency * Improve production efficiency * Develop cheaper and better products ## Government Big Data helps governments: * Understand citizens' needs * Combat fraud * Minimize system errors * Improve operations * Reduce costs * Improve public services * Increase efficiency --- # ⚡ Chapter 4 Ultra-Quick Revision ### Data Science **Data → Analysis → Patterns → Insights → Decisions** ### Data Types **Qualitative → Ordinal / Nominal** **Quantitative → Discrete / Continuous** ### Data Sources **Primary = Directly collected** **Secondary = Already collected by others** ### Big Data 3Vs **Volume = How much** **Velocity = How fast** **Variety = How many types** ### Big Data Challenges **Quality → Security → Growth → Tools → Integration** ### Big Data Applications **Healthcare → Media → IoT → Manufacturing → Government** ### Important Definitions to Memorize * Data * Data Analytics * Data Science * Dataset * Machine Learning * Deep Learning * Data Mining * Predictive Analysis * NLP * Database * DBMS * Primary Data * Secondary Data * Big Data * Predictive Maintenance * Data Visualization