Using a Driver List
dbc can create and manage lists of drivers using a driver list file.
By default, a driver list has the name dbc.toml, though this can be overridden.
Note
This functionality is similar to files from other tools such as Python's requirements.txt.
A driver list is ideal for checking into version control alongside your project and is useful for recording not only which drivers your project needs but also the specific versions of each.
Creating a Driver List
Create a driver list with dbc init:
Driver lists uses the TOML format and contains a TOML table of drivers. See the driver list reference for more detail.
Adding a Driver
While you can edit dbc.toml manually, dbc has subcommands for working with the list.
To add a driver to the list, use dbc add:
$ dbc add mysql
added mysql to driver list
use `dbc sync` to install the drivers in the list
$ cat dbc.toml
[drivers]
[drivers.mysql]
When run, the add command automatically checks that a driver matching the pattern exists in the driver registry and will fail if a matching driver can't be found.
Note
dbc add accepts the same syntax for driver names and versions as dbc install. See the Installing Drivers.
If you look closely at the above output, you'll notice that it's telling you to run dbc sync to install the driver(s) in the list. This is because dbc add only modifies the driver list and you need to use dbc sync to actually install the driver you just added.
Adding Multiple Drivers
You can add multiple drivers in a single command:
$ dbc add mysql snowflake
added mysql to driver list
added snowflake to driver list
use `dbc sync` to install the drivers in the list
$ cat dbc.toml
[drivers]
[drivers.mysql]
[drivers.snowflake]
Version constraints can be specified for each driver individually:
$ dbc add "mysql=0.1.0" "snowflake>=1.0.0"
added mysql to driver list with constraint =0.1.0
added snowflake to driver list with constraint >=1.0.0
use `dbc sync` to install the drivers in the list
Synchronizing
Use dbc sync to ensure that all the drivers in a driver list are installed:
The first time you run dbc sync, dbc creates a lockfile in the same directory as the driver list.
By default, this file is called dbc.lock but will match the name of your driver list file if you choose to use a custom one.
When you run dbc sync and a lockfile already exists, dbc will install the exact versions in the lockfile.
To upgrade the versions in the lockfile, delete the lockfile and run dbc sync.
Lockfile
dbc sync automatically creates a lockfile file in the same directory as the driver list. By default, this file is called dbc.lock but will match the name of your driver list file if you choose to use a custom one.
The lockfile records, for each driver, the exact version, the registry it came from, and the SHA-256 hash of the archive for each platform:
$ cat dbc.lock
# This file is automatically @generated by dbc. Not intended for manual editing
version = 2
[[drivers]]
name = 'mysql'
version = '0.1.0'
[drivers.source]
type = 'registry'
url = 'https://dbc-cdn.columnar.tech'
[[drivers.artifacts]]
format = 'tar.gz'
hash = 'sha256:...'
[drivers.artifacts.target]
os = 'linux'
arch = 'amd64'
libc = 'gnu'
[drivers.artifacts.location]
kind = 'registry-path'
value = 'mysql/v0.1.0/mysql_linux_amd64_v0.1.0.tar.gz'
[[drivers.artifacts]]
format = 'tar.gz'
hash = 'sha256:...'
[drivers.artifacts.target]
os = 'macos'
arch = 'arm64'
[drivers.artifacts.location]
kind = 'registry-path'
value = 'mysql/v0.1.0/mysql_macos_arm64_v0.1.0.tar.gz'
Each artifact records its archive format, SHA-256 hash, structured target, and location. A registry artifact uses kind = 'registry-path' with a path relative to the registry. An archive outside a registry uses a credential-free absolute URL; kind = 'path' is reserved for a future local source. An HTTPS registry cannot use an HTTP artifact URL.
When the registry publishes hashes, dbc sync records targets with valid, representable archive metadata. Unrecordable or ambiguous packages for other targets are omitted; a problem with the current target remains an error. For older releases without published hashes, only the current platform is recorded, using the hash of the archive it downloaded; another platform is added the first time dbc sync runs there. A target identifies its operating system and architecture. Linux targets use libc = 'gnu' by convention, while an explicit libc = 'musl' target is supported. This records the registry target convention; it does not change machine platform detection to inspect which C library is installed.
The lockfile can also carry an artifact size, source evidence, and host requirements supplied by a resolver. When an archive needs to be downloaded, dbc sync also checks its size if recorded. A matching installed archive can be reused without downloading, even when a size is recorded. Registry-generated locks currently omit size, evidence, and host requirements. Host requirements and evidence are receiver-side metadata only; their presence does not mean dbc sync enforces them.
When a lockfile exists, dbc sync uses the archive recorded for the current platform without consulting the registry index. A driver that is already installed is reused only when the installed package evidence matches that archive; signature verification is repeated against the installed library and signature before reuse unless --no-verify is set. If a lock file cannot be opened because of permissions or a read-only filesystem, dbc checks an optimistic read-only snapshot; observed changes prevent reuse. For a v1 lockfile, dbc checks the applicable library proof against the installed library or downloaded archive. If reuse checks fail, the recorded archive is downloaded, its hash is checked, and the driver is reinstalled. dbc downloads and checks every archive it needs before it writes the lockfile or changes any installed driver. A project lock serializes concurrent syncs through installation; acquiring it waits at most 10 seconds before returning an error. The new lockfile is written before installation starts, so if an installation then fails (for example on signature verification), the lockfile is kept and running dbc sync again retries with the same archives. --no-verify skips signature verification but never the archive hash check, and a driver installed with --no-verify is reinstalled with signature verification by a later dbc sync without that flag.
The lockfile never contains credentials. The registry is recorded without any user name, password or query string, and dbc sync uses your configured registry with the same credential-free identity to download archives. Configured registries that differ only by userinfo or query string have the same identity; if more than one matches a locked registry, dbc sync reports an ambiguous binding error. If the single matching registry URL includes a query string, dbc sync can't add a missing platform to an existing entry; delete the lockfile and run dbc sync again instead.
Lockfiles written by dbc versions before 0.4 (version = 1) are upgraded automatically: the locked version is kept as long as it still satisfies the driver list's version constraint, and the recorded checksum is still checked when you run dbc sync on the same platform. dbc 0.3.1 rejects a version 2 lockfile. dbc 0.3.0 and earlier ignore its version 2 fields, install the pinned driver names and versions without checking archive hashes, then silently rewrite the lockfile as version 1 and drop its version 2 metadata. All collaborators must upgrade to a dbc release that supports version 2 before sharing an upgraded lockfile.
Every time you run dbc sync, this file is updated with the exact information about each driver.
It's a good idea to track dbc.lock as well as dbc.toml in version control if you want to ensure a completely reproducible set of drivers.
Note
Unlike lockfiles in some other package managers, dbc.lock doesn't give each project its own isolated set of drivers. dbc sync installs drivers to the same location dbc install would (see Installing Drivers). If two projects on the same machine lock different versions of a driver, whichever project you sync last replaces the version the other one installed. To isolate a project's drivers, run dbc sync inside a Python virtual environment or conda environment, or set ADBC_DRIVER_PATH to a project-specific directory.
Version Constraints
Each driver in a driver list can optionally include a version constraint which dbc will respect when you run dbc sync. You can add a driver to the list with the same syntax as you used for dbc install, see Installing Drivers.
$ dbc add "mysql=0.1.0"
added mysql to driver list with constraint =0.1.0
use `dbc sync` to install the drivers in the list
$ cat dbc.toml
[drivers]
[drivers.mysql]
version = '=0.1.0'
Pre-release Versions
Allowing Pre-release Versions
By default, when you add a driver to a driver list, dbc will only consider stable (non-pre-release) versions. If you want to allow pre-release versions when running dbc sync, use the --pre flag with dbc add:
$ dbc add --pre mysql
added mysql to driver list
use `dbc sync` to install the drivers in the list
$ cat dbc.toml
[drivers]
[drivers.mysql]
prerelease = 'allow'
The prerelease = 'allow' field tells dbc sync to consider pre-release versions when resolving which version to install.
Note
The prerelease = 'allow' field only affects implicit version resolution. When your version constraint unambiguously references a pre-release (by including a pre-release suffix like -beta.1), that constraint will match pre-release versions regardless of the prerelease field.
Adding Specific Pre-release Versions
You can add a driver with a version constraint that references a specific pre-release version without using the --pre flag. When your version constraint unambiguously references a pre-release by including a pre-release suffix, the prerelease field is not added:
$ dbc add "mysql=1.0.0-beta.1"
added mysql to driver list with constraint =1.0.0-beta.1
use `dbc sync` to install the drivers in the list
$ cat dbc.toml
[drivers]
[drivers.mysql]
version = '=1.0.0-beta.1'
The version constraint =1.0.0-beta.1 unambiguously indicates you want a pre-release, so prerelease = 'allow' is not needed.
However, if your version constraint is ambiguous and only a pre-release version satisfies it, dbc sync will fail rather than install the pre-release. For example, if a driver has versions 0.1.0 and 0.1.1-beta.1:
To allow 0.1.1-beta.1 to be installed in this case, you must either:
- Use
dbc add --preto addprerelease = 'allow' - Change the constraint to reference the pre-release:
version = '>=0.1.1-beta.1'
Removing Drivers
Drivers can be removed from a driver list with the dbc remove command:
Using a Custom Filename
By default, dbc assumes a driver list has the filename dbc.toml. However, you can override this if you prefer another name or want to maintain multiple driver lists in one project (e.g., separate development and production lists).
All of the commands shown earlier on this page allow you to override the filename, for example: