2 posts tagged with "drift detection"

Hey everyone!

Welcome to the second Atlas release of 2025, v0.31! We're excited to share the latest updates and improvements with you. In this release you will find:

• Custom schema rules: You can now define custom rules for your database schema and have Atlas enforce them for you during CI.
• pgvector support: We've added support for managing schemas for projects that use the LLM-based pgvector extension.
• Drift detection: It is now simpler to set up drift detection checks to alert you when a target database isn't in the state it's supposed to be in.
• Multi-project ER Diagrams: you can now create composite ER diagrams that stitch schema objects from multiple Atlas projects.

Enforce custom schema rules​

atlas login

Atlas now supports the definition of custom schema rules that can be enforced during local development or CI/CD pipelines. This feature is particularly useful for teams that want to ensure that their database schema adheres to specific conventions or best practices, such as:

• "Require columns to be not null or have a default value"
• "Require foreign keys to reference primary keys of the target table and use ON DELETE CASCADE"
• "Require an index on columns used by foreign keys"
• ... and many more

The linted schema can be defined in any supported Atlas schema format, such as HCL, SQL, ORM, a URL to a database, or a composition of multiple schema sources.

Here's a simple example of a schema rule file:

# A predicate that checks if a column is not null or has a default value.
predicate "column" "not_null_or_have_default" {
  or {
    default {
      ne = null
    }
    null {
      eq = false
    }
  }
}

rule "schema" "disallow-null-columns" {
  description = "require columns to be not null or have a default value"
  table {
    column {
      assert {
        predicate = predicate.column.not_null_or_have_default
        message   = "column ${self.name} must be not null or have a default value"
      }
    }
  }
}

The example above defines a rule that checks if all columns in a schema are either not null or have a default value.

To use it, you would add the rule file to you atlas.hcl configuration:

lint {
  rule "hcl" "table-policies" {
    src = ["schema.rule.hcl"]
  }
}

env "local" {
  src = "file://schema.lt.hcl"
  dev = "sqlite://?mode=memory"
}

Suppose we used the following schema:

schema "main" {
}

table "hello" {
  schema = schema.main
  column "id" {
    type = int
    null = true
  }
}

Next, Atlas offers two ways to invoke the linting process:

1. Using the migrate lint command in the Atlas CLI. This lints only new changes that are introduced in the linted changeset - typically the new migration files in the current PR.
2. Using the newly added schema lint command in the Atlas CLI. This lints the entire schema, including all tables and columns, and can be used to enforce schema rules during CI/CD pipelines.

Let's use the schema lint command to lint the schema:

atlas schema lint --env local -u env://src

The command will output the following error:

Analyzing schema objects (3 objects in total):

  rule "disallow-null-columns":
    -- schema.lt.hcl:7: column id must be not null or have a default value

  -------------------------
  -- 27.791µs
  -- 1 diagnostic

Great! Atlas has detected that the id column in the hello table is nullable and has raised an error. You can now update the schema to fix the error and rerun the linting process.

For more information on defining custom schema rules, check the docs for custom schema rules.

Manage schemas for LLM-based projects with pgvector​

pgvector has become the de-facto standard for storing high-dimensional vectors in PostgreSQL. It is widely used in LLM-based projects that require fast similarity search for large datasets. Use cases such as recommendation systems and RAG (Retrieval-Augmented Generation) pipelines rely on pgvector to store embeddings and perform similarity searches.

In this release, we've added support for managing schemas for projects that use pgvector. You can now use Atlas to automate schema migrations for your pgvector projects.

Suppose you are building a RAG pipeline that uses OpenAI embeddings to generate responses to user queries. Your schema might look like this:

schema "public" {
}

extension "vector" {
  schema  = schema.public
}

table "content" {
  schema = schema.public
  column "id" {
    type = int
  }
  column "text" {
    type = text
  }
  column "embedding" {
    type = sql("vector(1536)")
  }
  primary_key {
    columns = [ column.id ]
  }
  index "idx_emb" {
    type = "IVFFlat"
    on {
      column = column.embedding
      ops    = "vector_l2_ops"
    }
    storage_params {
      lists = 100
    }
  }
}

Notice three key elements in the schema definition:

1. The extension block defines the vector extension, enabling pgvector support for the schema.
2. The column block defines a column named embedding with the vector(1536) type, which stores 1536-dimensional vectors.
3. The index block defines an index on the embedding column, which uses the IVFFlat method for similarity search.

For more information on defining pgvector powered indexes, check the docs for index storage parameters.

Simpler setup for Drift Detection​

Schema drift, which refers to a state where there is a difference between the intended database schema and its actual state, can lead to significant issues such as application errors, deployment failures, and even service outages.

In this release, we've simplified the process of setting up drift checks using the Atlas Cloud UI.

If you already use Atlas to run migrations, enable drift detection to stay on top of any manual schema changes that may cause a painful outage.

Watch the demo:

Multi-project ER Diagrams​

Many teams using Atlas adopt microservices-oriented architectures to build their applications. This approach allows them to develop, scale, and deploy services independently, but it also introduces a unique challenge: database schemas are often spread across multiple logical (and sometimes physical) databases.

While these architectures excel at modularity (by decoupling teams from one another) they make it harder to visualize, manage, and reason about the overall data model. Traditional Entity-Relationship (ER) diagrams are designed for monolithic databases, where all relationships and constraints exist within a single system. However, in a microservices setup, relationships between entities often span across services, sometimes without explicit foreign keys or enforced constraints.

To help teams overcome this challenge, we are happy to announce today the availability of composite, multi-project ER diagrams in Atlas Cloud. Using this feature, teams can create diagrams that are composed of database objects from multiple Atlas projects.

See it in action:

Related Object Selection​

Over years of development, database schemas tend to become complex and may grow to encompass hundreds or even thousands of objects. Using the Atlas data modeling tool, users can curate smaller subgraphs of the schema, named “saved filters”, which can serve as documentation for a certain aspect of the system. To make this process easier, our team has added a new context menu that makes it easier to include connected objects (e.g tables that are connected via a Foreign Key) in saved filters.

Watch the example:

New Trust Center​

At Ariga, the company behind Atlas, we are committed to meeting the evolving security and compliance needs of our customers. As part this commitment we've consolidated our legal and compliance documentation into a single page in our new Trust Center.

Wrapping Up​

We hope you enjoy the new features and improvements. As always, we would love to hear your feedback and suggestions on our Discord server.

Hi everyone,

Thanks for joining us today for another release announcement! We have a bunch of really exciting features to share with you today, so let's get started! Here's what we'll cover:

• Drift Detection - A common source of database trouble is that the schema in your database doesn't match the schema in your code. This can happen for a variety of reasons, including manual changes to the database, or changes made by other tools. Today, we are happy to announce the availability of a new feature that lets you automatically detect these changes, and alerts you when they happen.
• SQLAlchemy Support - SQLAlchemy is a popular Python ORM. Developers using SQLAlchemy can use Atlas to automatically plan schema migrations for them, based on the desired state of their schema instead of crafting them by hand.
• VSCode ERDs - We've added a new feature to our VSCode extension that lets you visualize your database schema as an ERD diagram.
• Composite Schemas - The newly added composite_schema data source lets you combine multiple schemas into one, which is useful for managing schemas that are loaded from multiple sources or to describe applications that span multiple database schemas.

Drift Detection​

We believe, that in an ideal world, schema migrations on production databases should be done in an automated way, preferably in your CI/CD pipelines, with developers not having root access. However, we know that this is oftentimes is not the case. For this reason, it is also common to find databases which schemas differ from the ones they are supposed to have. This phenomenon, called a Schema Drift can cause a lot of trouble for a team.

Atlas now can periodically check if your deployed databases schemas match their desired state. To function correctly, this feature relies on Atlas Cloud being able to communicate to your database. As it is uncommon for databases to be directly accessible from the internet, we have added the option to run Atlas Agents in your database's network to facilitate this communication. Agents register themselves via credentials against your Atlas Cloud account and continuously poll it for work.

To learn more about how to use this feature, check out our Drift Detection Guide.

In addition, Atlas Agents enable you do use a lot more cool features, like

• Cloud mediated deployments (coming soon)
• Schema monitoring and auditing (coming soon)

SQLAlchemy Support​

Goodbye, Alembic. Hello, Atlas.​

SQLAlchemy is a popular ORM toolkit widely used in the Python community. SQLAlchemy allows users to describe their data model using its declarative-mapping feature. To actually create the underlying tables, users can use the Base.metadata.create_all method which may be sufficient during development where tables can be routinely dropped and recreated.

However, at some point, teams need more control and decide to employ the versioned migrations methodology, which is a more robust way to manage a database schema.

The native way to manage migrations with SQLAlchemy is to use the Alembic migration tool. Alembic can automatically generate migration scripts from the difference between the current state of the database and the desired state of the application.

A downside of this approach is that in order for it to work, a pre-existing database with the current version of the schema must be connected to. In many production environments, databases should generally not be reachable from developer workstations, which means this comparison is normally done against a local copy of the database which may have undergone some changes that aren't reflected in the existing migrations.

In addition, Alembic auto-generation fails to detect many kinds of changes and cannot be relied upon to generate production-ready migration scripts without routine manual intervention.

Atlas, on the other hand, can automatically plan database schema migrations for SQLAlchemy without requiring a connection to such a database and can detect almost any kind of schema change. Atlas plans migrations by calculating the diff between the current state of the database, and its desired state.

To learn how to use Atlas with SQLAlchemy, check out our SQLAlchemy Guide.

Special thanks to No'am (Miko) Tamir (who also doubles as my young brother) for his fantastic work building the prototype for this feature and to Ronen Lubin for making it production-ready.

VSCode ERDs​

Starting with v0.4.2, our VSCode Extension can now visualize your database schema as an ERD diagram. To use this feature, simply open the command palette (Ctrl+Shift+P on Windows/Linux, Cmd+Shift+P on Mac) and select Atlas: View in ERD.

Composite Schemas​

The composite_schema data source allows the composition of multiple Atlas schemas into a unified schema graph. This functionality is useful when projects schemas are split across various sources such as HCL, SQL, or application ORMs. For example, each service might have its own schema.

Referring to the url returned by this data source allows reading the entire project schemas as a single unit by any of the Atlas commands, such as migrate diff, schema apply, or schema inspect.

Usage example​

By running atlas migrate diff with the given configuration, Atlas loads the inventory schema from the SQLAlchemy schema, the graph schema from ent/schema, and the auth and internal schemas from HCL and SQL schemas defined in Atlas format. Then, the composite schema, which represents these four schemas combined, will be compared against the current state of the migration directory. In case of a difference between the two states, a new migration file will be created with the necessary SQL statements.

data "composite_schema" "project" {
  schema "inventory" {
    url = data.external_schema.sqlalchemy.url
  }
  schema "graph" {
    url = "ent://ent/schema"
  }
  schema "auth" {
    url = "file://path/to/schema.hcl"
  }
  schema "internal" {
    url = "file://path/to/schema.sql"
  }
}

env "dev" {
  src = data.composite_schema.project.url
  dev = "docker://postgres/15/dev"
  migration {
    dir = "file://migrations"
  }
}

Wrapping up​

That's it! I hope you try out (and enjoy) all of these new features and find them useful. As always, we would love to hear your feedback and suggestions on our Discord server.