At DT, we serve ads at a scale where the data challenges are as real as the business challenges. Across our platform, teams were independently answering the same questions with different numbers because they were pulling from different tables, applying different transformations, and working from different assumptions. Upstream schema changes broke pipelines silently. Data scientists spent more time wrestling raw joins into shape than building models. 

Rather than patching individual issues and failures, we designed and built a shared logical foundation that every team, engineering, data science, analytics, and business users, works from. One definition of a user. One definition of a device. One way to calculate a device match rate or resolve a geographic hierarchy. Entities defined once, versioned, tested, and reused across every downstream system that needs them (Reporting, Data Science, Analytics, Engineering). The semantic layer that establishes a single source of truth by standardizing how core business entities are defined, built, and consumed. It ensures that analytics, machine learning models, and downstream applications all interpret data the same way. We’re still in the early stages of broad adoption, but the foundation is in place and teams are already building on top of it.

One of the clearest early signals of that payoff is in our audience pipelines. We feed semantic layer data as input to the pipeline that produces audience lists for users we bid on through our demand-side platform (DSP). The pipeline already had a solid feature set, but we wanted to give our per-app classifiers a richer signal about who a user actually is. With a clean, stable semantic foundation in place, we’re building dense user and app embeddings that encode behavioral and preference signals and feed them into those models as additional features. This work would have been far more challenging to do reliably against raw, ungoverned data.

This post covers what we built, how we built it, why we made the decisions we made, and what we’re learning along the way.

DT Semantic Layer: A Primer

Prior to the existence of the semantic layer, data at DT lived in silos. Each team curated and propagated its own datasets, but — more than that — the datasets lived across different buckets, different technologies, and different warehouses. The semantic layer we are building is driving consumers towards adopting a better option and working with a clear set of rules to build out downstream users’ dataset needs. Previously, even when the underlying raw data was the same, the interpretation rarely was.

This interpretation issue led to:

• Conflicting metrics across teams
• Repeated transformation logic
• Fragile pipelines and inconsistent definitions
• Reduced confidence in analytics and models
  

As part of the semantic layer design process, the first step taken was to gather a group of 15+ engineers, business intelligence SMEs, data scientists, and architects from across the company together to establish a data model and agree to foundational technologies used to implement that model. With the model agreed to, the Data Engineering Team set out to build the structure around which the dataset could be derived. Prior to those discussions and agreements, the Enterprise Data Team had already applied a commonly used data architecture pattern starting with a base medallion architecture and building up from there. Below is a view illustrating the overall architecture of the enterprise lakehouse for DT and where the semantic layer fits in around the Gold layer.

‍Building the DT Semantic Layer: A Phased Approach

We took a phased approach to building the semantic layer. First defining entities, then compiling entities into data pipelines based on data stored in the metadata registry, scheduling and running data pipeline jobs to create entities.

The DT Semantic Layer was built against a modern data lakehouse stack. Rather than adopting an off-the-shelf orchestration framework, we built purpose-specific layers on top of each technology to serve the specific demands of entity compilation and deployment:

• Databricks: We built a config-driven job factory that packages all entity ETL jobs into a single deployable Python wheel. Each entity gets its own Databricks job definition, and the factory instantiates the correct job at runtime from a shared registry meaning one wheel ships the entire set of semantic layer pipelines.
  
• Delta Lake: We built a three-strategy write system on top of Delta’s MERGE capabilities: full rebuild for reference tables, incremental merge for append-heavy entities, and a two-stage SCD2 merge for slowly changing dimensions (SCD). The SCD2 implementation in particular relies heavily on Delta’s ability to atomically match, update, and insert rows in a single operation.
  
• PySpark: We built a transform-chain ETL pattern where each pipeline step is a pure DataFrame → DataFrame function, as well as a foreign key resolution layer that handles deterministic joins across bridge tables without embedding that logic into individual jobs.
  
• Apache Airflow: We built auto-generated DAG tasks for each entity, with upstream and downstream dependencies wired at compile time based on the entity’s declared source relationships.
  
• Terraform: We built per-entity job provisioning so that adding a new entity to the semantic layer automatically provisions its compute resources, cluster configuration, and wheel reference in the production Databricks workspace.

Let’s step through the construction phases one by one and describe the pieces with examples.

Phase 1: Defining Entities

We created a separate repository for building and storing entity definitions apart from the compilation into data pipelines to separate concerns between logical and physical models. We followed the approach that this first phase would be a low- to no-code generation process using Claude as the backing agentic AI assistance technology. To keep costs lower and because we were satisfied with its behavioral characteristics, Sonnet 4.6 was used as the LLM. 

Our first step on the roadmap to automatically building entities was defining a CLAUDE.md file, the rulebook that tells Claude how to work within this project. The file includes Makefile targets for entity setup, validation, testing, and creation, along with schema conformity rules that verify generated pipeline code actually matches what the entity definition declares. That last piece matters more than it might seem: given an entity such as Device, an LLM asked to build the pipeline could silently hallucinate different relationships, wrong column mappings, or a mismatched grain. The conformity checks catch that drift by comparing generated output against the canonical entity YAML before anything reaches production.

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Commands

```bash
make setup-uv       # Create .venv and install deps (recommended, requires uv)
make setup-venv     # Create .venv and install deps (pip)
make validate       # Validate all entity files against schema + custom rules
make test           # Run pytest test suite
make lint           # Run flake8 on scripts/ and tests/
make test-watch     # Run pytest in watch mode
make status         # Show environment status
```

Run a single test:

```bash
.venv/bin/pytest tests/test_entities_conform_to_schema.py::TestRealEntities::test_all_entities_are_valid -v
```

Without make (after activating venv):

```bash
python scripts/validate_entities.py   # Validate entities
pytest tests/ -v                      # Run tests
```
...

‍Building Skills

Building skills came next. The agent needed a structured way to extract information before generating an entity. Essential questions — such as the entity’s domain, build strategy, and ownership — had to be answered upfront. Rather than relying on ad-hoc prompting, we built an add-entity-definition skill that codifies this questionnaire and drives the authoring process through Claude.

---
name: add-entity-definition
description: Add or edit semantic-layer entity definitions (YAML). Follow the entity implementation process: gather input, draft/apply, validate, run pre-merge checklist, and ask until 100% clear. Use when the user wants to add a new entity or change an existing one.
---

# Add entity definition

Use this skill when the user wants to **add a new entity** or **edit an existing entity** definition in the semantic layer. Follow the process below; do not guess — ask until everything is clear.

## When to use

- User says they want to **add** a new entity, create an entity definition, or add an entity definition for X.
- User says they want to **edit**, change, update, fix, or modify an existing entity; or has an `entities/**/entity.yaml` file open and asks for changes.

If the entity does not exist yet, use the **add** flow. If it already exists, use the **edit** flow.

---

Below is a table of skills with high-level descriptions that we built for the definition phase:

With skills and agent rules in place, the first phase became simply a matter of prompting. A single input such as “I want to define an entity named Creative” kicks off the agentic workflow. Claude then drives the conversation, asking the questions needed to capture the entity’s domain, relationships, build strategy, and structure. The result is a completed entity.yaml artifact.

From there, the author commits the entity YAML file, opens a pull request, and once merged to main, Phase 2 begins automatically: the CI pipeline verifies the entity definition and stores it in the metadata layer.

‍Phase 2: Building the Entity Metadata Registry 

The REST API handles write operations and CI/CD validation. But for the AI assistant Claude, we needed a different interface — the MCP.

We built an MCP server exposing exactly four tools:

MCP Features: Right Context For Right Audience

One of the most interesting design elements in the MCP layer is the ability to limit context based on who the audience is (who is interacting with the service). 

The default audience returns a public metadata summary: column names, types, nullability, descriptions, and a small number of representative sample values per column. Profiling statistics and internal implementation columns are omitted. The goal is a clean, concise overview suited to exploration and discovery.

The full audience must be requested explicitly. It returns the complete context: all columns, all profiling statistics, and the full sample set. This level of detail is appropriate when the output will drive code generation or query optimization — tasks that require null rates, cardinality estimates, and concrete value examples to make correct decisions.

The controlling point here is that the full audience is never the default. An interactive agent asking “what entities exist?” doesn’t need 60 columns × 10 profiling fields × 10 sample values per entity. That's hundreds of kilobytes of context the model doesn’t need and will likely misuse. The audience lens keeps token consumption proportional to the task. Keeping costs low in development required this segregation of duties between returning consumer based context.

def get_entities(
    names: list[str],
    audience: Audience,
):

When you call the discovery tool (list_entities), the response isn’t a flat list of names, it’s a graph. Consider a simplified response with three entities:

{ #-- example of how entities could tie to joins --
  "entities": [ #nodes
    { "name": "order","type": "fact","domain": "Commerce" },
    { "name": "customer","type": "dimension","domain": "Commerce"},
    { "name": "product",  "type": "dimension", "domain": "Catalog"}
  ],
  "joins": [ #edges
    { "from": "order", "via": "customer_id", "to": "customer" },
    { "from": "customer", "via": "customer_id", "to": "order" },
    { "from": "order", "via": "product_id",  "to": "product"  },
    { "from": "product", "via": "product_id",  "to": "order"  }
  ]
}

Because edges are emitted in both directions, an agent planning a query that starts at a customer can immediately see that the order is reachable, without scanning the full edge list looking for any edge whose field happens to be a customer. The join path is readable directly from the node’s neighbors (i.e., join → order to customer via customer_id).

An agent planning a multi-entity query can call list_entities once to discover the full graph, identify which entities it needs, then call get_entities once with all N names. Two calls total, regardless of how many entities exist. It’s about saving time and tokens through the full roundtrip.

Entity definitions live in a separate Git repository as mentioned above. When an engineer opens a pull request adding or modifying an entity, we want to validate the definition against the real Unity Catalog schema before merging so as not to discover that a referenced column doesn’t exist when the PySpark job runs in production.

‍MCP Features: Dry Run Entity Validation and Testing

The metadata registry supports an entity validation and testing feature on the generation endpoint (i.e., dry-run mode).

In dry-run mode, the engine performs the full harvest and strict-mode validation, confirming that the source table exists in our catalog and that every alias, column attribute reference resolves to a real column, but writes nothing to the context store. A non-empty failed array or a failed status fails the deployment pipeline.

‍MCP Features: Draft Entity Support

The service also supports “draft entities.” Engineers often need to merge an entity definition for review before its physical target table exists in Unity Catalog (often created by a later migration). By using the lifecycle: draft descriptor, the engine skips Unity Catalog introspection for the target table. It safely harvests sources, validates mappings, and writes the entity document without failing because the target does not yet exist.

The registry is the foundation for AI-driven semantic layer development. With it in place:

• A Claude session can call list_entities to understand the full topology, then get_entities on the entities it needs, and generate a correct multi-entity PySpark join with accurate column references and realistic filter values.
  
• A CI pipeline validates entity definitions against live schema on every pull request, catching column mismatches before they ever reach production.
  
• Entity context is always fresh. If a source table schema changes, the next entity read automatically includes the updated schema without regenerating the entity document.
  

The broader principle is that schema metadata is the infrastructure for AI‑assisted data engineering. Without it, AI assistants are working blind. With a well-structured registry, they have the context they need to generate code that runs.

Phase 3: Pipeline Creation and Deployment

A semantic definition on its own produces no data. The entity compiler agent is what closes that gap, transforming entity definitions into physical tables that live and are queryable in the lakehouse. An entity definition describes what a concept is: its attributes, its sources, and how it changes over time. The compiler takes that definition and generates everything needed to run it, the PySpark job, the Delta table schema, the Databricks compute configuration, the Airflow orchestration task, and the unit tests. By the time compilation is done, the entity is ready to deploy with no further engineering work.

The compiler translates the business entity design into six concrete, consistent artifacts for each entity:

Artifact: ETL job
File: jobs/build_<entity>_entity.py
Purpose: PySpark DataFrame transformations
────────────────────────────────────────

Artifact: Column model
File: model/<entity>.py
Purpose: Type-safe column constants (single contract)
────────────────────────────────────────

Artifact: Delta sink
File: delta/table/delta_<entity>_table.py
Purpose: Target schema and write strategy
────────────────────────────────────────

Artifact: Runtime config
File: resources/test.yaml, prod.yaml
Purpose: Source/sink bindings by environment
────────────────────────────────────────

Artifact: Terraform job
File: ci/terraform/environments/prod/terraform.tfvars
Purpose: Job provisioning
────────────────────────────────────────

Artifact: Airflow task
File: airflow/dags/.../build_semantic_layer.py
Purpose: Orchestration with upstream dependencies
────────────────────────────────────────

Keeping these artifacts consistent automatically rather than by convention was the core motivation for building the compiler. The two systems (entity design and compilation) can also evolve independently: entity definitions can gain new fields or profiling annotations without touching pipeline code; the compiler can upgrade its merge mechanics without touching entity semantics.

Compiling an Entity: What the Compiler Actually Does

When a new entity is compiled, it fetches the full definition from the entity MCP server and works through five phases in sequence:

‍Entity Build Strategies

Every entity declares a build strategy in its definition. The compiler generates different merge mechanics depending on which one is chosen:

Regression Testing

Even well-designed models drift. Upstream changes happen, new values appear, relationships break. Without something actively checking, you usually find out weeks later when someone questions a downstream result or report. The regression process we built rests on a simple truth: the data contract itself is what needs to be tested. A versioned set of SQL assertions covers the four things that actually break trust: uniqueness, referential integrity, valid values, and freshness. When a pipeline change silently drops a column, it surfaces as a test failure. When an entity relationship breaks, it shows up before it reaches a consumer. When a new entity ships, its tests ship with it, so coverage compounds rather than decays. The shift is from reactive fixes to preventative guardrails. Single source of truth only works if it stays true on every run.

A Note on CI/CD

The compilation process is what closes the loop between intent and execution. A data modeler defines what an entity is, the compiler generates everything needed to run it, and the deployment pipeline carries it the rest of the way. When a merge lands on main, the full test suite runs, the wheel is published to Google Cloud Storage, Terraform applies the updated Databricks job definitions to production, and the Airflow DAGs are deployed to Cloud Composer, all without a manual step. From that point, the entity is considered “published”. It runs on its scheduled cadence, its history is tracked, and its data is queryable in the lakehouse. The semantic layer was designed so that the distance between defining a business concept and having it reflected in production data is as short as possible and the compiler is what makes that distance manageable.

In Summary

Taken together, the three phases form a closed loop: a designer defines a business concept through a guided agentic workflow, the definition is validated and stored in the metadata registry, and the compiler transforms it into a fully deployed, tested, production pipeline with multiple human-in-the-loop review processes to enable any code to deploy into a production environment. The entity authoring process stays low-code and intent-driven; the compilation and deployment process is deterministic and auditable. The diagram below captures how these pieces connect.

The major features of the three-phased DT Semantic Layer build system are summarized in the following table to provide an overview:

The Results Are In: Data Science Team Sees 10‑20% Improvement Using Semantic Entities for Audience Embedding

An audience embedding is a way to turn user behaviors, traits, and contexts into a list of numbers (a vector) so machine learning models can easily understand them. Instead of grouping users into rigid, manually created categories (like “Age 25–34”), embeddings map users into a multi-dimensional mathematical space based on how they actually behave.

To evaluate the embedding features built from user–app interaction data sourced through the semantic layer, we used AUC (Area Under the Curve): a standard metric that measures how well a model separates users who will take a target action (install, in-app event, revenue) from those who won’t. An AUC of 0.5 is random guessing; 1.0 is perfect. Even small gains in the 0.70–0.90 range translate directly into better-targeted campaigns and higher return on ad spend (ROAS).

We tested across 6 apps and three model types, keeping all settings identical to production and adding only the embedding features. The results were clear:

• Post-install KPI models showed the largest gains (+0.09 to +0.15 AUC — 21% gain). App1 went from 0.72 to 0.87, App2 from 0.70 to 0.80. These are exactly the models where the existing feature set had the most room to grow, and where embeddings ranked #1 or #2 in feature importance.
  
• Revenue models improved by +0.09 to +0.14 AUC (20% gain).
  
• Install models started from a higher baseline (0.85–0.98), so gains were smaller (from +0.02  to +0.10 — 2%10% gain) there’s simply less headroom for growth.

Driving Adoption

A semantic layer only succeeds if teams actually use it. We focused on three primary levers:

• Visibility: Exposing these entities through a Data Catalog (Unity Catalog) makes the entities easy to discover, understand, and utilize in production capacity. The Data Catalog serves as the discovery and governance backbone for the semantic layer.
• Proof of Value: Demonstrating improved performance and consistency in data science models and analytics. Previously, data science teams relied on legacy datasets built outside the semantic layer process. Once they migrated to the new semantic entities, their model results improved significantly, yielding a 10% to 20% performance increase across various use cases. 
• Adoption Measurement: Building agents to create adoption measurement statistics from data lakehouse logs and downstream (non-semantic layer) job sources. This provides the transparency needed to establish where improvements need to be made.

By aligning both engineering and data science workflows around shared definitions, we’ve seen improvements in the data science team’s ability to build embeddings for audiences. We are actively working cross functionally to continue driving adoption by making it easier for downstream consumers (think reporting, BI, and product analytics).

Closing Thoughts and Considerations

The DT Enterprise Data Platform touches billions of user records and active devices. At that scale, a misapplied filter or wrong join doesn’t produce a mildly inaccurate report, it produces a confidently wrong one, and decisions about which campaigns to fund, which markets to grow, and which device segments to target get made against it. The cost of ambiguous data isn't theoretical; it compounds quietly across every team that builds on top of it.

The semantic layer is how we address that at the source. The geo entity means a location isn’t recorded as “India” for one team, “IN” for another, and “IND” for a third — it is a standardized identifier used universally across all teams. Similarly, the device entity means analyzing Android versus iOS performance no longer requires querying messy, raw manufacturer strings; instead, teams simply join on clean, unified identifiers. It is a complete standardization of how data across DT is connected and extended. These aren’t cosmetic improvements. They’re the difference between data that is merely available and data that is trustworthy.

What we’ve built is more than a data modeling initiative. Treating entity definitions as versioned, tested, executable assets — and using AI-driven automation to make authoring them fast and consistent — changes the relationship between data producers and consumers across the organization. Teams stop rebuilding the same concepts independently and start building on a shared foundation. The semantic layer is how we're ensuring that as DT’s scale grows, so does our confidence in the data underneath it.