\r

Data Evolution Analysis\r

\r

Overview\r

\r Based on DDC methodology (Chapter 1.1), this skill analyzes data evolution patterns in construction organizations, assessing digital maturity levels from paper-based workflows to fully data-driven operations.\r \r Book Reference: "Эволюция использования данных в строительной отрасли" / "Evolution of Data Usage in Construction"\r \r

Quick Start\r

\r

from dataclasses import dataclass, field\r
from enum import Enum\r
from typing import List, Dict, Optional\r
from datetime import datetime\r
import json\r
\r
class MaturityLevel(Enum):\r
    """Digital maturity levels based on DDC methodology"""\r
    LEVEL_0_PAPER = 0      # Paper-based, no digital tools\r
    LEVEL_1_BASIC = 1      # Basic digital (spreadsheets, email)\r
    LEVEL_2_STRUCTURED = 2  # Structured databases, some integration\r
    LEVEL_3_INTEGRATED = 3  # ERP/BIM integration, workflows\r
    LEVEL_4_AUTOMATED = 4   # Automated processes, ML/AI\r
    LEVEL_5_PREDICTIVE = 5  # Predictive analytics, digital twins\r
\r
class DataCategory(Enum):\r
    """Categories of construction data"""\r
    DESIGN = "design"\r
    COST = "cost"\r
    SCHEDULE = "schedule"\r
    QUALITY = "quality"\r
    SAFETY = "safety"\r
    PROCUREMENT = "procurement"\r
    DOCUMENT = "document"\r
    COMMUNICATION = "communication"\r
\r
@dataclass\r
class DataFlowAssessment:\r
    """Assessment of data flow in an organization"""\r
    category: DataCategory\r
    source_systems: List[str]\r
    storage_format: str\r
    integration_level: float  # 0-1\r
    automation_level: float   # 0-1\r
    data_quality_score: float # 0-1\r
    issues: List[str] = field(default_factory=list)\r
\r
@dataclass\r
class MaturityAssessment:\r
    """Complete digital maturity assessment"""\r
    organization_name: str\r
    assessment_date: datetime\r
    overall_level: MaturityLevel\r
    category_scores: Dict[DataCategory, float]\r
    data_flows: List[DataFlowAssessment]\r
    strengths: List[str]\r
    weaknesses: List[str]\r
    recommendations: List[str]\r
    roadmap: Dict[str, List[str]]\r
\r
\r
class DataEvolutionAnalyzer:\r
    """\r
    Analyze data evolution and digital maturity in construction organizations.\r
    Based on DDC methodology Chapter 1.1.\r
    """\r
\r
    def __init__(self):\r
        self.assessment_criteria = self._load_criteria()\r
        self.evolution_stages = self._define_evolution_stages()\r
\r
    def _load_criteria(self) -> Dict[DataCategory, Dict]:\r
        """Load assessment criteria for each category"""\r
        return {\r
            DataCategory.DESIGN: {\r
                "tools": ["CAD", "BIM", "Collaboration Platform"],\r
                "metrics": ["model_usage", "clash_detection", "design_reviews"],\r
                "weight": 0.20\r
            },\r
            DataCategory.COST: {\r
                "tools": ["Spreadsheets", "Estimating Software", "ERP"],\r
                "metrics": ["automation_level", "historical_data", "benchmarking"],\r
                "weight": 0.15\r
            },\r
            DataCategory.SCHEDULE: {\r
                "tools": ["Gantt Charts", "CPM Software", "4D BIM"],\r
                "metrics": ["resource_loading", "progress_tracking", "forecasting"],\r
                "weight": 0.15\r
            },\r
            DataCategory.QUALITY: {\r
                "tools": ["Checklists", "QC Software", "Defect Tracking"],\r
                "metrics": ["inspection_digitization", "defect_analytics", "compliance"],\r
                "weight": 0.12\r
            },\r
            DataCategory.SAFETY: {\r
                "tools": ["Incident Reports", "Safety Software", "IoT Sensors"],\r
                "metrics": ["incident_tracking", "predictive_safety", "training"],\r
                "weight": 0.12\r
            },\r
            DataCategory.PROCUREMENT: {\r
                "tools": ["RFQ Manual", "e-Procurement", "Supply Chain"],\r
                "metrics": ["vendor_management", "material_tracking", "integration"],\r
                "weight": 0.10\r
            },\r
            DataCategory.DOCUMENT: {\r
                "tools": ["File Shares", "DMS", "CDE"],\r
                "metrics": ["version_control", "access_control", "searchability"],\r
                "weight": 0.08\r
            },\r
            DataCategory.COMMUNICATION: {\r
                "tools": ["Email", "Collaboration", "Unified Platform"],\r
                "metrics": ["response_time", "transparency", "audit_trail"],\r
                "weight": 0.08\r
            }\r
        }\r
\r
    def _define_evolution_stages(self) -> Dict[MaturityLevel, Dict]:\r
        """Define characteristics of each evolution stage"""\r
        return {\r
            MaturityLevel.LEVEL_0_PAPER: {\r
                "name": "Paper-Based",\r
                "description": "Manual, paper-based processes",\r
                "characteristics": [\r
                    "Physical document storage",\r
                    "Manual data entry",\r
                    "Limited data sharing",\r
                    "No real-time visibility"\r
                ],\r
                "typical_tools": ["Paper forms", "Physical filing"]\r
            },\r
            MaturityLevel.LEVEL_1_BASIC: {\r
                "name": "Basic Digital",\r
                "description": "Basic digitization with standalone tools",\r
                "characteristics": [\r
                    "Spreadsheets for calculations",\r
                    "Email for communication",\r
                    "File shares for storage",\r
                    "Manual data transfer between systems"\r
                ],\r
                "typical_tools": ["Excel", "Word", "Email", "File shares"]\r
            },\r
            MaturityLevel.LEVEL_2_STRUCTURED: {\r
                "name": "Structured Data",\r
                "description": "Structured databases and specialized software",\r
                "characteristics": [\r
                    "Department-specific software",\r
                    "Structured databases",\r
                    "Basic reporting",\r
                    "Some standardization"\r
                ],\r
                "typical_tools": ["CAD", "Estimating software", "Project software"]\r
            },\r
            MaturityLevel.LEVEL_3_INTEGRATED: {\r
                "name": "Integrated Systems",\r
                "description": "Connected systems with data flow",\r
                "characteristics": [\r
                    "ERP integration",\r
                    "BIM adoption",\r
                    "Automated workflows",\r
                    "Cross-department data sharing"\r
                ],\r
                "typical_tools": ["BIM", "ERP", "CDE", "BI dashboards"]\r
            },\r
            MaturityLevel.LEVEL_4_AUTOMATED: {\r
                "name": "Automated & Analytics",\r
                "description": "Automation and advanced analytics",\r
                "characteristics": [\r
                    "Automated data collection",\r
                    "Machine learning models",\r
                    "Predictive analytics",\r
                    "Real-time dashboards"\r
                ],\r
                "typical_tools": ["ML platforms", "IoT", "Advanced analytics"]\r
            },\r
            MaturityLevel.LEVEL_5_PREDICTIVE: {\r
                "name": "Predictive & Autonomous",\r
                "description": "AI-driven, predictive operations",\r
                "characteristics": [\r
                    "Digital twins",\r
                    "Autonomous decision support",\r
                    "Continuous optimization",\r
                    "Predictive maintenance"\r
                ],\r
                "typical_tools": ["Digital twins", "AI/ML", "Autonomous systems"]\r
            }\r
        }\r
\r
    def assess_organization(\r
        self,\r
        organization_name: str,\r
        survey_responses: Dict[str, any],\r
        system_inventory: List[Dict],\r
        process_documentation: Optional[Dict] = None\r
    ) -> MaturityAssessment:\r
        """\r
        Perform comprehensive digital maturity assessment.\r
\r
        Args:\r
            organization_name: Name of the organization\r
            survey_responses: Responses from maturity survey\r
            system_inventory: List of systems/tools in use\r
            process_documentation: Optional process documentation\r
\r
        Returns:\r
            Complete maturity assessment\r
        """\r
        # Analyze data flows\r
        data_flows = self._analyze_data_flows(system_inventory, survey_responses)\r
\r
        # Calculate category scores\r
        category_scores = self._calculate_category_scores(\r
            data_flows, survey_responses\r
        )\r
\r
        # Determine overall maturity level\r
        overall_score = sum(\r
            score * self.assessment_criteria[cat]["weight"]\r
            for cat, score in category_scores.items()\r
        )\r
        overall_level = self._score_to_level(overall_score)\r
\r
        # Identify strengths and weaknesses\r
        strengths, weaknesses = self._identify_gaps(category_scores)\r
\r
        # Generate recommendations\r
        recommendations = self._generate_recommendations(\r
            overall_level, weaknesses, data_flows\r
        )\r
\r
        # Create roadmap\r
        roadmap = self._create_roadmap(overall_level, recommendations)\r
\r
        return MaturityAssessment(\r
            organization_name=organization_name,\r
            assessment_date=datetime.now(),\r
            overall_level=overall_level,\r
            category_scores=category_scores,\r
            data_flows=data_flows,\r
            strengths=strengths,\r
            weaknesses=weaknesses,\r
            recommendations=recommendations,\r
            roadmap=roadmap\r
        )\r
\r
    def _analyze_data_flows(\r
        self,\r
        system_inventory: List[Dict],\r
        survey_responses: Dict\r
    ) -> List[DataFlowAssessment]:\r
        """Analyze data flows between systems"""\r
        flows = []\r
\r
        for category in DataCategory:\r
            # Find systems for this category\r
            category_systems = [\r
                s for s in system_inventory\r
                if s.get("category") == category.value\r
            ]\r
\r
            if not category_systems:\r
                flows.append(DataFlowAssessment(\r
                    category=category,\r
                    source_systems=[],\r
                    storage_format="none",\r
                    integration_level=0.0,\r
                    automation_level=0.0,\r
                    data_quality_score=0.0,\r
                    issues=["No systems identified for this category"]\r
                ))\r
                continue\r
\r
            # Analyze integration and automation\r
            integration = self._calculate_integration_score(category_systems)\r
            automation = self._calculate_automation_score(\r
                category_systems, survey_responses\r
            )\r
            quality = survey_responses.get(\r
                f"{category.value}_data_quality", 0.5\r
            )\r
\r
            # Identify issues\r
            issues = self._identify_flow_issues(\r
                category_systems, integration, automation\r
            )\r
\r
            flows.append(DataFlowAssessment(\r
                category=category,\r
                source_systems=[s["name"] for s in category_systems],\r
                storage_format=category_systems[0].get("format", "unknown"),\r
                integration_level=integration,\r
                automation_level=automation,\r
                data_quality_score=quality,\r
                issues=issues\r
            ))\r
\r
        return flows\r
\r
    def _calculate_integration_score(\r
        self, systems: List[Dict]\r
    ) -> float:\r
        """Calculate integration score for systems"""\r
        if not systems:\r
            return 0.0\r
\r
        total_integrations = sum(\r
            len(s.get("integrations", [])) for s in systems\r
        )\r
        max_integrations = len(systems) * 3  # Assume max 3 integrations per system\r
\r
        return min(1.0, total_integrations / max_integrations)\r
\r
    def _calculate_automation_score(\r
        self,\r
        systems: List[Dict],\r
        survey: Dict\r
    ) -> float:\r
        """Calculate automation score"""\r
        scores = []\r
\r
        for system in systems:\r
            system_score = 0.0\r
            if system.get("has_api"):\r
                system_score += 0.3\r
            if system.get("automated_imports"):\r
                system_score += 0.3\r
            if system.get("automated_exports"):\r
                system_score += 0.2\r
            if system.get("workflow_automation"):\r
                system_score += 0.2\r
            scores.append(system_score)\r
\r
        return sum(scores) / len(scores) if scores else 0.0\r
\r
    def _calculate_category_scores(\r
        self,\r
        data_flows: List[DataFlowAssessment],\r
        survey: Dict\r
    ) -> Dict[DataCategory, float]:\r
        """Calculate maturity score for each category"""\r
        scores = {}\r
\r
        for flow in data_flows:\r
            # Combine different aspects\r
            tool_score = survey.get(f"{flow.category.value}_tool_maturity", 0.5)\r
            process_score = survey.get(f"{flow.category.value}_process_maturity", 0.5)\r
\r
            category_score = (\r
                tool_score * 0.3 +\r
                process_score * 0.2 +\r
                flow.integration_level * 0.2 +\r
                flow.automation_level * 0.2 +\r
                flow.data_quality_score * 0.1\r
            )\r
\r
            scores[flow.category] = category_score\r
\r
        return scores\r
\r
    def _score_to_level(self, score: float) -> MaturityLevel:\r
        """Convert numeric score to maturity level"""\r
        if score \x3C 0.1:\r
            return MaturityLevel.LEVEL_0_PAPER\r
        elif score \x3C 0.25:\r
            return MaturityLevel.LEVEL_1_BASIC\r
        elif score \x3C 0.45:\r
            return MaturityLevel.LEVEL_2_STRUCTURED\r
        elif score \x3C 0.65:\r
            return MaturityLevel.LEVEL_3_INTEGRATED\r
        elif score \x3C 0.85:\r
            return MaturityLevel.LEVEL_4_AUTOMATED\r
        else:\r
            return MaturityLevel.LEVEL_5_PREDICTIVE\r
\r
    def _identify_gaps(\r
        self,\r
        scores: Dict[DataCategory, float]\r
    ) -> tuple[List[str], List[str]]:\r
        """Identify strengths and weaknesses"""\r
        avg_score = sum(scores.values()) / len(scores)\r
\r
        strengths = [\r
            f"{cat.value}: {score:.0%}"\r
            for cat, score in scores.items()\r
            if score > avg_score + 0.1\r
        ]\r
\r
        weaknesses = [\r
            f"{cat.value}: {score:.0%}"\r
            for cat, score in scores.items()\r
            if score \x3C avg_score - 0.1\r
        ]\r
\r
        return strengths, weaknesses\r
\r
    def _identify_flow_issues(\r
        self,\r
        systems: List[Dict],\r
        integration: float,\r
        automation: float\r
    ) -> List[str]:\r
        """Identify issues in data flow"""\r
        issues = []\r
\r
        if integration \x3C 0.3:\r
            issues.append("Low system integration - data silos likely")\r
        if automation \x3C 0.3:\r
            issues.append("Manual data transfer required")\r
        if len(systems) > 3:\r
            issues.append("Multiple overlapping systems")\r
\r
        return issues\r
\r
    def _generate_recommendations(\r
        self,\r
        level: MaturityLevel,\r
        weaknesses: List[str],\r
        flows: List[DataFlowAssessment]\r
    ) -> List[str]:\r
        """Generate improvement recommendations"""\r
        recommendations = []\r
\r
        # Level-specific recommendations\r
        level_recs = {\r
            MaturityLevel.LEVEL_0_PAPER: [\r
                "Implement basic digital tools (spreadsheets, file sharing)",\r
                "Digitize critical paper-based processes",\r
                "Train staff on basic digital skills"\r
            ],\r
            MaturityLevel.LEVEL_1_BASIC: [\r
                "Adopt specialized construction software",\r
                "Implement structured data storage",\r
                "Standardize data formats and naming conventions"\r
            ],\r
            MaturityLevel.LEVEL_2_STRUCTURED: [\r
                "Integrate key systems (ERP, PM, BIM)",\r
                "Implement Common Data Environment (CDE)",\r
                "Develop automated workflows"\r
            ],\r
            MaturityLevel.LEVEL_3_INTEGRATED: [\r
                "Implement advanced analytics and dashboards",\r
                "Explore IoT for automated data collection",\r
                "Develop machine learning models for prediction"\r
            ],\r
            MaturityLevel.LEVEL_4_AUTOMATED: [\r
                "Implement digital twin technology",\r
                "Deploy AI-driven decision support",\r
                "Enable predictive maintenance and operations"\r
            ],\r
            MaturityLevel.LEVEL_5_PREDICTIVE: [\r
                "Continuous optimization of AI models",\r
                "Expand autonomous decision-making",\r
                "Industry leadership and knowledge sharing"\r
            ]\r
        }\r
\r
        recommendations.extend(level_recs.get(level, []))\r
\r
        # Address specific weaknesses\r
        for flow in flows:\r
            if flow.integration_level \x3C 0.3:\r
                recommendations.append(\r
                    f"Improve {flow.category.value} system integrations"\r
                )\r
            if flow.data_quality_score \x3C 0.5:\r
                recommendations.append(\r
                    f"Implement data quality controls for {flow.category.value}"\r
                )\r
\r
        return recommendations[:10]  # Top 10 recommendations\r
\r
    def _create_roadmap(\r
        self,\r
        current_level: MaturityLevel,\r
        recommendations: List[str]\r
    ) -> Dict[str, List[str]]:\r
        """Create phased improvement roadmap"""\r
        return {\r
            "Phase 1 (0-6 months)": recommendations[:3],\r
            "Phase 2 (6-12 months)": recommendations[3:6],\r
            "Phase 3 (12-24 months)": recommendations[6:],\r
            "Target Level": [\r
                f"Move from {current_level.name} to "\r
                f"{MaturityLevel(min(current_level.value + 1, 5)).name}"\r
            ]\r
        }\r
\r
    def compare_assessments(\r
        self,\r
        assessments: List[MaturityAssessment]\r
    ) -> Dict:\r
        """Compare multiple assessments over time or across organizations"""\r
        comparison = {\r
            "assessments": len(assessments),\r
            "levels": [a.overall_level.name for a in assessments],\r
            "trends": {},\r
            "best_practices": []\r
        }\r
\r
        # Track category trends\r
        for category in DataCategory:\r
            scores = [a.category_scores[category] for a in assessments]\r
            comparison["trends"][category.value] = {\r
                "scores": scores,\r
                "improvement": scores[-1] - scores[0] if len(scores) > 1 else 0\r
            }\r
\r
        return comparison\r
\r
    def generate_report(\r
        self,\r
        assessment: MaturityAssessment\r
    ) -> str:\r
        """Generate executive summary report"""\r
        stage_info = self.evolution_stages[assessment.overall_level]\r
\r
        report = f"""\r
# Digital Maturity Assessment Report\r
## {assessment.organization_name}\r
\r
**Assessment Date:** {assessment.assessment_date.strftime('%Y-%m-%d')}\r
**Overall Maturity Level:** {assessment.overall_level.name} - {stage_info['name']}\r
\r
### Executive Summary\r
{stage_info['description']}\r
\r
### Category Scores\r
"""\r
        for cat, score in assessment.category_scores.items():\r
            bar = "█" * int(score * 10) + "░" * (10 - int(score * 10))\r
            report += f"- {cat.value.title()}: {bar} {score:.0%}\
"\r
\r
        report += "\
### Strengths\
"\r
        for strength in assessment.strengths:\r
            report += f"- {strength}\
"\r
\r
        report += "\
### Areas for Improvement\
"\r
        for weakness in assessment.weaknesses:\r
            report += f"- {weakness}\
"\r
\r
        report += "\
### Recommendations\
"\r
        for i, rec in enumerate(assessment.recommendations, 1):\r
            report += f"{i}. {rec}\
"\r
\r
        report += "\
### Roadmap\
"\r
        for phase, items in assessment.roadmap.items():\r
            report += f"\
**{phase}**\
"\r
            for item in items:\r
                report += f"- {item}\
"\r
\r
        return report\r
\r
\r
class DataEvolutionTracker:\r
    """Track data evolution over time"""\r
\r
    def __init__(self, organization_name: str):\r
        self.organization = organization_name\r
        self.history: List[MaturityAssessment] = []\r
        self.milestones: List[Dict] = []\r
\r
    def add_assessment(self, assessment: MaturityAssessment):\r
        """Add new assessment to history"""\r
        self.history.append(assessment)\r
        self._check_milestones(assessment)\r
\r
    def _check_milestones(self, assessment: MaturityAssessment):\r
        """Check if any milestones were reached"""\r
        if len(self.history) > 1:\r
            prev = self.history[-2]\r
\r
            # Level improvement\r
            if assessment.overall_level.value > prev.overall_level.value:\r
                self.milestones.append({\r
                    "date": assessment.assessment_date,\r
                    "type": "level_up",\r
                    "description": f"Advanced from {prev.overall_level.name} "\r
                                   f"to {assessment.overall_level.name}"\r
                })\r
\r
            # Category improvements\r
            for cat in DataCategory:\r
                if assessment.category_scores[cat] - prev.category_scores[cat] > 0.2:\r
                    self.milestones.append({\r
                        "date": assessment.assessment_date,\r
                        "type": "category_improvement",\r
                        "description": f"Significant improvement in {cat.value}"\r
                    })\r
\r
    def get_evolution_summary(self) -> Dict:\r
        """Get summary of evolution over time"""\r
        if not self.history:\r
            return {"error": "No assessments recorded"}\r
\r
        return {\r
            "organization": self.organization,\r
            "first_assessment": self.history[0].assessment_date,\r
            "latest_assessment": self.history[-1].assessment_date,\r
            "starting_level": self.history[0].overall_level.name,\r
            "current_level": self.history[-1].overall_level.name,\r
            "total_assessments": len(self.history),\r
            "milestones": self.milestones,\r
            "level_progression": [a.overall_level.value for a in self.history]\r
        }\r
```\r
\r
## Common Use Cases\r
\r
### Assess Current Digital Maturity\r
\r
```python\r
analyzer = DataEvolutionAnalyzer()\r
\r
# Define systems in use\r
systems = [\r
    {"name": "AutoCAD", "category": "design", "has_api": False},\r
    {"name": "Revit", "category": "design", "has_api": True, "integrations": ["Navisworks"]},\r
    {"name": "Excel", "category": "cost", "has_api": False},\r
    {"name": "MS Project", "category": "schedule", "has_api": False},\r
    {"name": "Email", "category": "communication", "has_api": False}\r
]\r
\r
# Survey responses (from questionnaire)\r
survey = {\r
    "design_tool_maturity": 0.6,\r
    "design_process_maturity": 0.5,\r
    "design_data_quality": 0.7,\r
    "cost_tool_maturity": 0.3,\r
    "cost_process_maturity": 0.4,\r
    "cost_data_quality": 0.5,\r
    "schedule_tool_maturity": 0.4,\r
    "schedule_process_maturity": 0.3,\r
    "schedule_data_quality": 0.4\r
}\r
\r
assessment = analyzer.assess_organization(\r
    organization_name="Construction Co",\r
    survey_responses=survey,\r
    system_inventory=systems\r
)\r
\r
print(f"Maturity Level: {assessment.overall_level.name}")\r
print(f"Recommendations: {assessment.recommendations[:3]}")\r
```\r
\r
### Track Evolution Over Time\r
\r
```python\r
tracker = DataEvolutionTracker("Construction Co")\r
\r
# Add quarterly assessments\r
tracker.add_assessment(q1_assessment)\r
tracker.add_assessment(q2_assessment)\r
tracker.add_assessment(q3_assessment)\r
\r
summary = tracker.get_evolution_summary()\r
print(f"Progress: {summary['starting_level']} → {summary['current_level']}")\r
print(f"Milestones: {len(summary['milestones'])}")\r
```\r
\r
### Generate Executive Report\r
\r
```python\r
report = analyzer.generate_report(assessment)\r
print(report)\r
\r
# Save to file\r
with open("maturity_report.md", "w") as f:\r
    f.write(report)\r
```\r
\r
## Quick Reference\r
\r
| Component | Purpose |\r
|-----------|---------|\r
| `DataEvolutionAnalyzer` | Main assessment engine |\r
| `MaturityLevel` | 6 levels from paper to predictive |\r
| `DataCategory` | 8 categories (design, cost, schedule, etc.) |\r
| `DataFlowAssessment` | Analyze data flows per category |\r
| `MaturityAssessment` | Complete assessment results |\r
| `DataEvolutionTracker` | Track progress over time |\r
\r
## Resources\r
\r
- **Book**: "Data-Driven Construction" by Artem Boiko, Chapter 1.1\r
- **Website**: https://datadrivenconstruction.io\r
\r
## Next Steps\r
\r
- Use [data-silo-detection](../../Chapter-1.2/data-silo-detection/SKILL.md) to identify integration gaps\r
- Use [erp-integration-analysis](../../Chapter-1.2/erp-integration-analysis/SKILL.md) for system integration\r
- Use [digital-maturity-assessment](../../Chapter-5.1/digital-maturity-assessment/SKILL.md) for detailed assessments\r