Skip to main content

Automate Database CI/CD with GitLab Pipelines

GitLab CI is an automation tool integrated into GitLab that helps streamline code testing, building, and deployment through configurable pipelines. Using a .gitlab-ci.yml file, developers define jobs and stages that GitLab executes automatically, enabling consistent integration and delivery workflows.

Atlas provides CI/CD components for easy integration in any project. Click here to see the full list of components in the CI/CD catalog.

Check out this guide for full examples of the versioned and declarative flows using the GitLab components.

info

To use the components in a self-hosted instance, you must mirror the component project.

Offline Access​

Atlas GitLab CI components include built-in grant caching for offline continuity by storing .atlas as cache/artifacts and restoring it in subsequent jobs and pipelines.

No additional cache setup is required when using the official Atlas components.

migrate-push​

Pushes a migration directory to the cloud.

Usage​

services:
  - mysql:latest

variables:
  MYSQL_DATABASE: dev
  MYSQL_ROOT_PASSWORD: pass

stages:
  - push

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/migrate-push@~latest
    inputs:
      stage: push
      dir: 'file://migrations'
      dir-name: 'my-project'
      dev-url: 'mysql://root:pass@mysql:3306/dev'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • stage - (Optional) The CI stage to run the job in.
  • branches - The branches to run this jobs on.
  • working-directory - (Optional) Atlas working directory, default is project root
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev-databases.
  • dir - The URL of the migration directory to lint. For example: file://migrations. Read more about Atlas URLs here: https://atlasgo.io/concepts/url.
  • dir-name - (Optional) The name (slug) of the project in Atlas Cloud.
  • tag - (Optional) The tag to apply to the pushed migration directory. By default the current git commit hash is used.
  • latest - (Optional) If true, push also the "latest" tag.
  • atlas-cloud-token - A token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

migrate-lint​

Use this component to lint new migration files in any merge request.

Usage​

services:
  - mysql:latest

variables:
  MYSQL_DATABASE: dev
  MYSQL_ROOT_PASSWORD: pass

stages:
  - lint

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/migrate-lint@~latest
    inputs:
      stage: lint
      dir: 'file://migrations'
      dir-name: 'my-project'
      dev-url: 'mysql://root:pass@mysql:3306/dev'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root.
  • stage - (Optional) The CI stage to run the job in.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about here.
  • dir - (Optional) The URL of the migration directory to lint. For example: file://migrations. Read more about Atlas URLs here: https://atlasgo.io/concepts/url.
  • dir-name: The name (slug) of the project in Atlas Cloud.
  • atlas-cloud-token- (Optional) A token for Atlas Cloud.
  • atlas-version: The version of Atlas to install.
  • gitlab-token- (Optional) A GitLab access token for posting merge request comments.

migrate-apply​

Use this component to deploy migration files to your database directly from your CI runner. You can either deploy a local directory (i.e from your git repository) or a directory stored in Atlas Cloud.

info

Atlas needs network access to your database to deploy migrations, so make sure your database is either publicly accessible or that you have otherwise running this component from a runner that has network access to your database.

Usage​

Deploy a directory from the git repository​

stages:
  - apply

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/migrate-apply@~latest
    inputs:
      stage: apply
      dir: 'file://migrations'
      url: $DATABASE_URL

Deploy a directory from the cloud​

stages:
  - apply

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/migrate-apply@~latest
    inputs:
      stage: apply
      dir: 'atlas://my-project'
      url: $DATABASE_URL
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • branches - The branches to run this jobs on.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • url - (Optional) The URL of the target database to apply changes to. For example: mysql://root:pass@localhost:3306/prod.
  • dir - The URL of the migration directory to apply. For example: atlas://dir-name for cloud based directories or file://migrations for local ones.
  • dry-run - (Optional) If true, print SQL without executing it.
  • revisions-schema - (Optional) The name of the schema containing the revisions table.
  • atlas-cloud-token - (Optional) A token for Atlas Cloud.
  • atlas-version - Install a specific Atlas version.

migrate-diff​

Use this component to automatically generate versioned migrations whenever the schema is changed, and commit them to the migration directory.

Usage​

stages:
  - diff

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/migrate-diff@~latest
    inputs:
      stage: migrate-diff
      dir: 'file://migrations'
      to: 'file://schema.hcl'
      dev-url: 'mysql://root:pass@mysql:3306/dev'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN
      gitlab-token: $GITLAB_TOKEN

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • dir - (Optional) The URL of the migration directory to apply. For example: atlas://dir-name for cloud based directories or file://migrations for local ones.
  • to - (Optional) The URL of the desired schema state to transition to. For example: file://schema.hcl.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about here.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • remote - (Optional) The remote to push changes to. Defaults to origin.
  • gitlab-token - A Gitlab access token to push changes to the remote.
  • atlas-cloud-token - A token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

schema-plan​

Creates a plan for a declarative schema change. Read more about pre-planning schema changes here.

To enable Atlas comments on merge requests, you need create an access token with appropriate role and permissions, set it as a CI/CD variable and pass it to the component in the "gitlab-token" input. See the usage example below:

Usage​

services:
  - mysql:latest

variables:
  MYSQL_DATABASE: dev
  MYSQL_ROOT_PASSWORD: pass

stages:
  - plan

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/schema-plan@~latest
    inputs:
      stage: plan
      env: dev
      dev-url: 'mysql://root:pass@mysql:3306/dev'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN
      gitlab-token: $GITLAB_TOKEN

Inputs​

  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev-databases.
  • dir - The URL of the migration directory to lint. For example: file://migrations. Read more about Atlas URLs here: https://atlasgo.io/concepts/url.
  • dir-name - (Optional) The name (slug) of the project in Atlas Cloud.
  • tag - (Optional) The tag to apply to the pushed migration directory. By default the current git commit hash is used.
  • latest - (Optional) If true, push also the "latest" tag.
  • atlas-cloud-token - A token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.
  • gitlab-token- (Optional) A GitLab access token for posting merge request comments.

schema-plan-approve​

Approves a pending plan in Atlas Cloud for a specific change (i.e. the diff between the current and desired states). Run this job on the main branch of your project to automatically approve plans that were generated by the schema-plan job in previous merge requests.

If more than one pending plan is found for a given change, the job will exit with a non-zero code. In this case you will need to Choose the prefered plan and delete the others, before re-runnig the job.

Usage​

stages:
  - push

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/schema-plan-approve@~latest
    inputs:
      stage: push
      env: gitlab
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • branches - The branches to run this jobs on.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev urls here: https://atlasgo.io/concepts/dev-database.
  • schema-name - (Optional) The name (slug) of the schema repository in Atlas Registry. Read more in Atlas website: Atlas Registry.
  • from - (Optional) URL(s) of the current schema state.
  • to - (Optional) URL(s) of the desired schema state.
  • plan - (Optional) The URL of the plan to be approved. For example, atlas://<schema>/plans/<id>. If not provided, Atlas will search the registry for a plan corresponding to the given schema transition and approve it (typically, this plan is created during the PR stage). If multiple plans are found, an error will be thrown.
  • atlas-cloud-token - A token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

schema-push​

Pushes a declarative schema to the cloud.

Usage​

stages:
  - push

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/schema-push@~latest
    inputs:
      stage: push
      env: dev
      latest: true
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • branches - The branches to run this jobs on.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev urls here: https://atlasgo.io/concepts/dev-database.
  • schema-name - (Optional) The name (slug) of the schema repository in Atlas Registry. Read more in Atlas website: Atlas Registry.
  • url - (Optional) Desired schema URL(s) to push. For example: file://schema.hcl.
  • schema - (Optional) List of database schema(s). For example: public.
  • description - (Optional) The description of the schema.
  • latest - (Optional) If true, push also to the "latest" tag.
  • atlas-cloud-token - A token for Atlas Cloud.
  • atlas-version - (Optional) a specific Atlas version.

schema-apply​

Use this component to apply a schema to your database directly from your CI runner. You can either deploy a local state (i.e from your git repository) or a schema stored in Atlas Cloud.

info

Atlas needs network access to your database to deploy migrations, so make sure your database is either publicly accessible or that you have otherwise running this component from a runner that has network access to your database.

Usage​

stages:
  - apply

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/schema-apply@~latest
    inputs:
      stage: apply
      env: dev
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN # Needed only for deploying a schema from Atlas Cloud.

Inputs​

  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • branches - The branches to run this jobs on.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev urls here: https://atlasgo.io/concepts/dev-database.
  • schema-name - (Optional) The name (slug) of the schema repository in Atlas Registry. Read more in Atlas website: Atlas Registry.
  • to - (Optional) URL(s) of the desired schema state.
  • url - (Optional) The URL of the target database to apply changes to. For example: mysql://root:pass@localhost:3306/prod.
  • schema - (Optional) List of database schema(s). For example: public.
  • dry-run - (Optional) If true, print SQL without executing it.
  • auto-approve - (Optional) If true, automatically approve and apply changes.
  • plan - (Optional) The plan to apply. For example, atlas://<schema>/plans/<id>.
  • atlas-cloud-token - (Optional) A token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

monitor-schema​

Monitor changes of the database schema and track them in Atlas Cloud. Read more about schema monitoring here.

info

Atlas needs network access to your database to monitor schema changes, so make sure your database is either publicly accessible or that you have otherwise running this component from a runner that has network access to your database.

Usage​

stages:
  - monitor-schema

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/monitor-schema@~latest
    inputs:
      stage: monitor-schema
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN
      url: $DB_URL
      slug: gitlab-monitor

To run on a schedule, go to the Build tab in your repository, Go to the Pipeline Schedules tab and click on New Schedule.

Inputs​

  • atlas-cloud-token - A token for Atlas Cloud for reporting changes.
  • working-directory - (Optional) Atlas working directory, default is project root
  • stage - (Optional) The CI stage to run the job in.
  • url - URL of the database to Monitor (mutually exclusive with config and env).
  • config - The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects (mutually exclusive with url).
  • env - The environment to use from the Atlas configuration file. For example, dev (mutually exclusive with url) .
  • slug - (Optional) Unique identifier for the database server.
  • schemas - (Optional) List of database schema(s). For example: public.
  • exclude - (optional) List of exclude patterns from inspection. see.
  • include - (optional) List of include patterns from inspection. see.
  • collect-stats - (optional) Collect table statistics. Defaults to true.

Data Scripts Components​

Data Scripts define data operations as code: transactional mutations (script "exec"), reads and reports (script "query"), and batched work (script "loop"). The components below let you test scripts on merge requests, publish them to the Atlas Registry, and run them from a pipeline, on a schedule or manually.

Atlas Pro Feature

Data Scripts are available to Atlas Pro users that purchased Atlas Pipelines. Pass the atlas-cloud-token input to all of the components below so the job logs in before running.

All five components accept files (or paths, for script-test) pointing at the script source, and can instead read it from the selected environment in your Atlas configuration file, via the config and env inputs:

atlas.hcl
env "prod" {
  url = getenv("DATABASE_URL")
  script {
    src = "file://scripts"
    repo {
      name = "my-scripts"
    }
  }
  test {
    script {
      src = ["scripts.test.hcl"]
    }
  }
}
caution

Unlike migrate-lint or monitor-schema, the script components do not define rules of their own, so their job runs in every pipeline. Scope it explicitly: redefine the job by its job_name in your .gitlab-ci.yml, and the configuration is merged into the one the component contributes.

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-exec@~latest
    inputs:
      job_name: purge-users
      stage: run
      url: $DATABASE_URL
      files: 'file://scripts'
      match: '^purge_users$'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

# Run the mutation only on a schedule, or manually from the pipeline view.
purge-users:
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: on_success
    - when: manual

Outputs​

The script components report their outputs as a dotenv artifact, so downstream jobs that needs the script job read them as environment variables. Their names follow ATLAS_OUTPUT_<COMPONENT>_<OUTPUT>, for example:

notify:
  stage: notify
  needs:
    - job: atlas-script-query
      artifacts: true
  script:
    - echo "$ATLAS_OUTPUT_SCRIPT_QUERY_OUTPUT"

script-test​

Run tests for your Data Scripts. Use it on merge requests to verify a script's effect on the database, its expected failures, and the permissions it runs under, before it ever touches production data.

Usage​

services:
  - postgres:latest

variables:
  POSTGRES_DB: dev
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: pass

stages:
  - test

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-test@~latest
    inputs:
      stage: test
      paths: 'file://scripts'
      dev-url: 'postgres://postgres:pass@postgres/dev?sslmode=disable'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • job_name - (Optional) The name of the job. Defaults to atlas-script-test.
  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • paths - (Optional) List of directories containing test files.
  • run - (Optional) Filter tests to run by regexp. For example, ^test_.* will only run tests that start with test_. Default is to run all tests.
  • dev-url - (Optional) The URL of the dev-database to use for analysis. For example: mysql://root:pass@localhost:3306/dev. Read more about dev urls here: https://atlasgo.io/concepts/dev-database.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • atlas-cloud-token - A bot token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

script-push​

Pushes your Data Scripts to the Atlas Registry. Once pushed, pipelines and deployment tools can run them by name with an atlas://<name> URL, instead of checking the script source out on every runner.

Usage​

stages:
  - push

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-push@~latest
    inputs:
      stage: push
      script-name: 'my-scripts'
      files: 'file://scripts'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Inputs​

  • job_name - (Optional) The name of the job. Defaults to atlas-script-push.
  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • script-name - (Optional) The name (slug) of the script repository in Atlas Registry. Can also be set with env.script.repo.name in the configuration file.
  • files - (Optional) URL(s) of the script files or directories to push. For example: file://scripts. If not set, Atlas uses the scripts defined in the Atlas configuration file.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • atlas-cloud-token - A bot token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

Outputs​

  • ATLAS_OUTPUT_SCRIPT_PUSH_FILES - The number of script files that were pushed.
  • ATLAS_OUTPUT_SCRIPT_PUSH_LINK - Link to the scripts in Atlas Registry.
  • ATLAS_OUTPUT_SCRIPT_PUSH_NAME - The name of the script repository that was pushed to.

script-exec​

Execute script "exec" transactional mutations, such as a backfill, an archive, or a GDPR erasure, on a target database. The statements of each script run in a single transaction, guarded by its condition, assert, and check blocks, and a failing guard rolls the whole batch back.

The job fails if the run fails, or if any script in it fails, including a failed assert or check. An unmet condition stops its script gracefully and is not treated as a failure.

info

Atlas needs network access to your database to run scripts, so make sure your database is either publicly accessible or that you have otherwise running this component from a runner that has network access to your database.

Usage​

stages:
  - run

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-exec@~latest
    inputs:
      stage: run
      url: $DATABASE_URL
      files: 'file://scripts'
      # Anchor the pattern, otherwise `erase_user` also selects `erase_user_v2`.
      match: '^erase_user$'
      vars: '{"user_id": "$USER_ID"}'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Instead of checking the scripts out, point files at a repository pushed to the registry, for example files: 'atlas://my-scripts'. To run the mutation on demand, define the pipeline with input variables such as USER_ID above, and scope the job with rules as described above.

Inputs​

  • job_name - (Optional) The name of the job. Defaults to atlas-script-exec.
  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • url - (Optional) The URL of the target database to run the scripts against. For example: mysql://root:pass@localhost:3306/prod.
  • files - (Optional) URL(s) of the script files or directories to run. For example: file://scripts. If not set, Atlas uses the scripts defined in the Atlas configuration file.
  • match - (Optional) Run only the scripts matching the given regexp. For example, ^purge_.* will only run scripts that start with purge_. Default is to run all scripts. Note that the pattern is not anchored, and that a pattern matching nothing runs no scripts and succeeds.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • atlas-cloud-token - A bot token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

Outputs​

  • ATLAS_OUTPUT_SCRIPT_EXEC_OUTPUT - The output of the executed scripts, joined by newlines. It contains the values emitted by their output blocks and the results of their query blocks.
  • ATLAS_OUTPUT_SCRIPT_EXEC_REPORT - A JSON report of the run, containing the result of each executed script.

script-query​

Run script "query" scripts on a target database and collect what they print. Use it for scheduled reports and data-health checks, with sensitive columns masked before they leave Atlas.

Usage​

stages:
  - report

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-query@~latest
    inputs:
      stage: report
      url: $DATABASE_URL
      files: 'file://scripts'
      match: '^data_health$'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

To run the report periodically, go to the Build tab in your repository, go to the Pipeline Schedules tab and click on New Schedule.

Inputs​

  • job_name - (Optional) The name of the job. Defaults to atlas-script-query.
  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • url - (Optional) The URL of the target database to run the scripts against. For example: mysql://root:pass@localhost:3306/prod.
  • files - (Optional) URL(s) of the script files or directories to run. For example: file://scripts. If not set, Atlas uses the scripts defined in the Atlas configuration file.
  • match - (Optional) Run only the scripts matching the given regexp. For example, ^report_.* will only run scripts that start with report_. Default is to run all scripts.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • atlas-cloud-token - A bot token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

Outputs​

  • ATLAS_OUTPUT_SCRIPT_QUERY_OUTPUT - The output of the executed scripts, joined by newlines. It contains the values emitted by their output blocks and the results of their query blocks.
  • ATLAS_OUTPUT_SCRIPT_QUERY_REPORT - A JSON report of the run, containing the result of each executed script.

script-loop​

Run script "loop" scripts, which process rows in batches, on a target database. Each iteration runs in its own transaction, so a long purge or backfill makes progress without holding a single, long-lived transaction open.

info

Atlas needs network access to your database to run scripts, so make sure your database is either publicly accessible or that you have otherwise running this component from a runner that has network access to your database.

Usage​

stages:
  - run

include:
  - component: $CI_SERVER_FQDN/arigaio/atlas/script-loop@~latest
    inputs:
      stage: run
      url: $DATABASE_URL
      files: 'file://scripts'
      match: '^purge_inactive$'
      atlas-cloud-token: $ATLAS_CLOUD_TOKEN

Long-running purges may exceed the job timeout of your runner. Bound the work each run does with the loop's batch size and limits, and let a pipeline schedule pick the rest up on the next run.

Inputs​

  • job_name - (Optional) The name of the job. Defaults to atlas-script-loop.
  • stage - (Optional) The CI stage to run the job in.
  • working-directory - (Optional) Atlas working directory, default is project root
  • url - (Optional) The URL of the target database to run the scripts against. For example: mysql://root:pass@localhost:3306/prod.
  • files - (Optional) URL(s) of the script files or directories to run. For example: file://scripts. If not set, Atlas uses the scripts defined in the Atlas configuration file.
  • match - (Optional) Run only the scripts matching the given regexp. For example, ^purge_.* will only run scripts that start with purge_. Default is to run all scripts.
  • env - (Optional) The environment to use from the Atlas configuration file. For example, dev.
  • config - (Optional) The path to the Atlas configuration file. See: https://atlasgo.io/atlas-schema/projects.
  • vars - (Optional) A JSON object containing variables to be used in the Atlas configuration file. For example, {"var1": "value1", "var2": "value2"}.
  • atlas-cloud-token - A bot token for Atlas Cloud.
  • atlas-version - (Optional) Install a specific Atlas version.

Outputs​

  • ATLAS_OUTPUT_SCRIPT_LOOP_OUTPUT - The output of the executed scripts, joined by newlines. It contains the values emitted by their output blocks and the results of their query blocks.
  • ATLAS_OUTPUT_SCRIPT_LOOP_REPORT - A JSON report of the run, containing the result of each executed script.