Programmatic invocations

Programmatic invocations let you call dbt commands from Python scripts and applications, instead of running them in a shell. This is useful when you want to embed dbt runs into a larger application or workflow, while still using the same command surface area as the dbt Core CLI.

Common use cases include:

• Running dbt as part of a Python application or service
• Integrating dbt runs into orchestration workflows
• Building internal tools that need to run dbt commands and inspect results

Refer to the dbt Core package on PyPI to install the official Python package for dbt Core if you haven't done so already.

(Applies to dbt v2.0 and later)

from dbt.cli.main import dbtRunner, dbtRunnerResult

# initialize
dbt = dbtRunner()

# create CLI args as a list of strings
cli_args = ["run", "--select", "tag:my_tag"]

# run the command
res: dbtRunnerResult = dbt.invoke(cli_args)

# inspect the results
for r in res.result:
    print(f"{r.unique_id}: {r.status}")

For implementation details, refer to the dbt-python crate in the dbt Core repository.

Supported arguments

dbtRunner.invoke accepts the same arguments as the dbt Core CLI. The first positional argument is the command (for example, run, build, test), followed by any flags and options you would normally pass on the command line.

For example, dbt.invoke(["run", "--select", "tag:my_tag"]) is equivalent to running dbt run --select tag:my_tag. There is no separate, dbtRunner‑specific list of arguments; the authoritative source for available options is the CLI help reference (dbt --help, dbt run --help, and so on) and the dbt command reference documentation.

from dbt.cli.main import dbtRunner
dbt = dbtRunner()
# equivalent ways to pass arguments
dbt.invoke(["run", "--select", "tag:my_tag"])
dbt.invoke(["run"], select="tag:my_tag")

Parallel execution not supported

dbt-core doesn't support safe parallel execution for multiple invocations in the same process. Running multiple dbt commands concurrently in one process is unsafe and officially discouraged, and requires a wrapping process to manage subprocesses. This is because:

• Running concurrent commands can unexpectedly interact with the data platform. For example, running dbt run and dbt build for the same models simultaneously could lead to unpredictable results.
• Each dbt-core command interacts with global Python variables. To ensure safe operation, commands need to be executed in separate processes, for example by spawning subprocesses or using Celery for orchestration.

For safe parallel execution, you can use the dbt CLI or Studio IDE, both of which do that additional work to manage concurrency (multiple processes) on your behalf.

(Applies to dbt v2.0 and later)

In v2, invocations are serialized through thread-level locks, so multiple invocations can't run concurrently within the same process. (In v1, parallel execution was unsupported but there was no locking, so invocations could still run in multithreaded mode.) As in v1, you can still parallelize by using multiprocessing to run each invocation in a separate process.

dbtRunnerResult

Each command returns a dbtRunnerResult object with the following attributes:

• success (bool): Whether the command succeeded.
• result: When the command completes (successfully or with handled errors), it returns the command's result(s). The return type varies by command.
• exception: When the dbt invocation encounters an unhandled error and does not complete, the exception that was raised.
• catalog (v2 only): The catalog that the command produces when you request catalog generation.

(Applies to dbt v2.0 and later)

The v2 engine is implemented in Rust, so exception no longer contains the exact Python exception object raised by dbt. Instead, the caught error message is forwarded under an exception type:

• If the invocation fails at a foreign function interface (FFI) boundary before the engine picks up the invocation, exception contains an unwrapped exception type, such as ValueError or RuntimeError.
• If the invocation fails inside the engine, exception is a DbtRunnerError.

v2 also adds a top-level catalog attribute to dbtRunnerResult when catalog generation is requested.

In v1, catalog.json was only created when you ran dbt docs generate. In v2, you can generate the catalog as part of any command by passing the --write-catalog flag. For example, dbt run --write-catalog populates both dbtRunnerResult.result and dbtRunnerResult.catalog.

There is a one-to-one correspondence between CLI exit codes and the dbtRunnerResult returned by a programmatic invocation:

Scenario	CLI Exit Code	success	result	exception
Invocation completed without error	0	True	varies by command	None
Invocation completed with at least one handled error (for example, test failure or model build error)	1	False	varies by command	None
Unhandled error. Invocation did not complete, and returns no results.	2	False	None	Exception

Commitments and caveats

We're making an ongoing commitment to providing a Python entry point at functional parity with dbt Core's CLI. We reserve the right to change the underlying implementation used to achieve that goal. We expect that the current implementation will unlock real use cases in the short- and medium-term while we work on a set of stable, long-term interfaces that will ultimately replace it.

In particular, the objects returned by each command in dbtRunnerResult.result are not fully contracted, and therefore liable to change. Some of the returned objects are partially documented, because they overlap in part with the contents of dbt artifacts. As Python objects, they contain many more fields and methods than what's available in the serialized JSON artifacts. These additional fields and methods should be considered internal and liable to change in future versions of dbt-core.

Advanced usage patterns

caution

The syntax and support for these patterns are liable to change in future versions of dbt-core.

The goal of dbtRunner is to offer parity with CLI workflows within a programmatic environment. There are a few advanced usage patterns that extend what's possible with the CLI.

(Applies to dbt v2.0 and later)

Reusing objects

Manifest injection isn't supported in v2. You can't pass a pre-constructed Manifest into dbtRunner.

Registering callbacks

Registering callbacks on dbt's EventManager isn't supported in v2.

Overriding parameters

Pass in parameters as keyword arguments, instead of a list of CLI-style strings. At present, dbt will not do any validation or type coercion on your inputs. The command must be specified, in a list, as the first positional argument.

from dbt.cli.main import dbtRunner
dbt = dbtRunner()

# these are equivalent
dbt.invoke(["--fail-fast", "run", "--select", "tag:my_tag"])
dbt.invoke(["run"], select=["tag:my_tag"], fail_fast=True)

Was this page helpful?

Privacy policyCreate a GitHub issue

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.