Developer Tools

Migration

# Migration

Preflight and run reviewed live copies, portable bundle exports, and resumable bundle imports from the Varc CLI.

Varc runs foreground, one-shot migrations between Redis-compatible targets or
through a portable migration bundle. Every operation is planned before writes,
uses dedicated connections, and produces a complete, incomplete, or
indeterminate terminal result.

Migration planning and execution require a Pro or Team CLI Token with the
one-shot migration capability.

## Choose a migration mode

| Mode            | Reads from        | Writes to      | Use it for                                      |
| --------------- | ----------------- | -------------- | ----------------------------------------------- |
| `live_copy`     | Source connection | Destination    | Direct foreground copy between reachable hosts. |
| `bundle_export` | Source connection | `.varcmigrate` | Portable, offline, or staged transfer.          |
| `bundle_import` | `.varcmigrate`    | Destination    | Verified import with optional checkpoints.      |

Definitions live under `[migrations.<name>]` in `varc.toml`. Each definition
has its own `schema_version = 1` and strict mode-specific fields.

## Plan, review, then run

Planning connects and performs mandatory preflight without copying keys:

```bash
export VARC_CLI_TOKEN='varc_cli_...'
varc migrate plan tenant_copy
```

Write a versioned, non-secret plan artifact when the review and execution need
to be separated:

```bash
varc migrate plan tenant_copy \
  --plan-output artifacts/tenant-copy.plan.json

varc migrate run --plan artifacts/tenant-copy.plan.json
```

The run verifies the plan digest, redials both endpoints, repeats preflight,
and refuses changed stable endpoint evidence before writing. Plan and run each
obtain their own short-lived purpose-specific grant.

`run` is the explicit write intent. There is no generic `--yes` flag. Concrete
risks such as raw bundle export and excessive clock skew use named
acknowledgements instead.

## Live copy

Use `live_copy` when both endpoints are reachable for the full operation:

```toml
[connections.source]
url_env = "SOURCE_REDIS_URL"

[connections.destination]
url_env = "DESTINATION_REDIS_URL"

[migrations.tenant_copy]
schema_version = 1
mode = "live_copy"
source = "source"
destination = "destination"

[migrations.tenant_copy.scope]
key_type = "hash"
matcher = { kind = "prefix", encoding = "text", value = "tenant:" }

[migrations.tenant_copy.policy]
conflict = "stop"
expiration = "keep_expiry_time"
verification = "quick"
allow_compatibility_skips = false

[migrations.tenant_copy.execution]
max_keys_per_second = 5000
timeout = "2h"

[migrations.tenant_copy.output]
progress = "auto"
progress_interval = "10s"
problems_file = "artifacts/tenant-copy.problems.jsonl"
```

```bash
varc migrate plan tenant_copy
varc migrate run tenant_copy
```

The same named connection can be the source and destination only when the
selected databases differ. Cluster databases must be `0`.

## Export a portable bundle

Bundle export reads from one source and writes a `.varcmigrate` file:

```toml
[migrations.tenant_export]
schema_version = 1
mode = "bundle_export"
source = "source"
bundle_path = "artifacts/tenant.varcmigrate"
overwrite = "refuse"
estimated_uncompressed_bytes = 10737418240
minimum_free_after_export_bytes = 1073741824
source_labels = ["production", "pre-cutover"]

[migrations.tenant_export.scope]
matcher = { kind = "glob", encoding = "text", value = "tenant:*" }

[migrations.tenant_export.safety]
raw_export_acknowledged = true

[migrations.tenant_export.output]
progress = "plain"
problems_file = "artifacts/tenant-export.problems.jsonl"
```

```bash
varc migrate plan tenant_export
varc migrate run tenant_export
```

`raw_export_acknowledged` confirms that the portable bundle contains raw data.
The free-space reserve is checked in addition to the conservative uncompressed
size estimate. Existing bundles are refused unless overwrite is explicitly set
to `replace_validated_regular_file`.

## Import with a checkpoint

Bundle import validates the bundle before writing to the destination:

```toml
[migrations.tenant_import]
schema_version = 1
mode = "bundle_import"
bundle_path = "artifacts/tenant.varcmigrate"
destination = "destination"

[migrations.tenant_import.policy]
conflict = "stop"
expiration = "keep_expiry_time"
verification = "quick"
allow_compatibility_skips = false

[migrations.tenant_import.execution]
max_keys_per_second = 5000
timeout = "4h"

[migrations.tenant_import.output]
progress = "auto"
progress_file = "artifacts/tenant-import.progress.jsonl"
problems_file = "artifacts/tenant-import.problems.jsonl"
checkpoint_file = "artifacts/tenant-import.checkpoint.json"
```

```bash
varc migrate plan tenant_import
varc migrate run tenant_import
```

The versioned checkpoint records completed bundle chunks for a later reviewed
import attempt. It does not turn an interrupted operation into a success or
authorize blind replay of an indeterminate write.

## Scope and policy

A migration scope can select one key type and one matcher; when both are set,
they use AND semantics. Matchers support binary-safe prefix, suffix, and glob
values encoded as text, hex, or Base64, plus a UTF-8 regex:

```toml
matcher = { kind = "prefix", encoding = "text", value = "tenant:" }
matcher = { kind = "suffix", encoding = "hex", value = "00ff" }
matcher = { kind = "glob", encoding = "base64", value = "dGVuYW50Oio=" }
matcher = { kind = "regex", pattern = "^tenant:\\x00[0-9]+$" }
```

For live copy and import, policy controls conflicts, expiry handling,
verification, and compatibility skips:

- `conflict = "stop"`, `"skip"`, or `"replace"`;
- `expiration = "keep_expiry_time"` or `"restart_captured_ttl"`; and
- `verification = "quick"` or `"none"`.

Failed writes and verification failures are always incomplete. Conflict skips
are successful only under `conflict = "skip"`. Compatibility skips remain
incomplete unless `allow_compatibility_skips` is explicitly enabled.

## Progress and output

Visible progress is written to stderr; the final human or JSON result remains
the only stdout output. `progress = "auto"` selects a one-line TTY display in
an interactive terminal and plain periodic heartbeats in CI. You can also
choose `tty`, `plain`, `jsonl`, or `none`.

```bash
varc --non-interactive --format json migrate run tenant_copy \
  --progress jsonl \
  --progress-file artifacts/tenant-copy.progress.jsonl
```

Named migrations accept operational overrides such as progress paths and
intervals, timeout, output replacement, checkpoint path, and the named safety
acknowledgements. They reject target, mode, scope, and policy flags so a
reviewed definition cannot be silently redirected. Omit the migration name and
use `varc migrate --help` for the equivalent ad-hoc definition flags.

Problems files are bounded and binary-safe and never contain values or DUMP
payloads. Existing output files are replaced only when replacement is enabled;
symlinks are refused.

## Interpret the result safely

Exit code `0` means the migration completed under the selected policy. Code `5`
means the aggregate result is incomplete or indeterminate. Do not blindly retry
an indeterminate write: review the terminal result, problems file, and any
import checkpoint before deciding how to continue.

Clock skew above the engine threshold must be acknowledged with
`clock_skew_acknowledged = true` or `--acknowledge-clock-skew`. This gate applies
to live copy and bundle import. Raw-data acknowledgement applies only to bundle
export.

## Next steps

- [Set up the CLI](/docs/varc-cli/getting-started)
- [Read the complete TOML reference](/docs/varc-cli/configuration)
- [Build and run benchmarks](/docs/varc-cli/benchmark)
- [Validate and run scripts](/docs/varc-cli/scripts)

Preflight and run reviewed live copies, portable bundle exports, and resumable bundle imports from the Varc CLI.

Varc runs foreground, one-shot migrations between Redis-compatible targets or through a portable migration bundle. Every operation is planned before writes, uses dedicated connections, and produces a complete, incomplete, or indeterminate terminal result.

Migration planning and execution require a Pro or Team CLI Token with the one-shot migration capability.

Choose a migration mode

Mode	Reads from	Writes to	Use it for
`live_copy`	Source connection	Destination	Direct foreground copy between reachable hosts.
`bundle_export`	Source connection	`.varcmigrate`	Portable, offline, or staged transfer.
`bundle_import`	`.varcmigrate`	Destination	Verified import with optional checkpoints.

Definitions live under [migrations.<name>] in varc.toml. Each definition has its own schema_version = 1 and strict mode-specific fields.

Plan, review, then run

Planning connects and performs mandatory preflight without copying keys:

export VARC_CLI_TOKEN='varc_cli_...'
varc migrate plan tenant_copy

Write a versioned, non-secret plan artifact when the review and execution need to be separated:

varc migrate plan tenant_copy \
  --plan-output artifacts/tenant-copy.plan.json

varc migrate run --plan artifacts/tenant-copy.plan.json

The run verifies the plan digest, redials both endpoints, repeats preflight, and refuses changed stable endpoint evidence before writing. Plan and run each obtain their own short-lived purpose-specific grant.

run is the explicit write intent. There is no generic --yes flag. Concrete risks such as raw bundle export and excessive clock skew use named acknowledgements instead.

Live copy

Use live_copy when both endpoints are reachable for the full operation:

[connections.source]
url_env = "SOURCE_REDIS_URL"

[connections.destination]
url_env = "DESTINATION_REDIS_URL"

[migrations.tenant_copy]
schema_version = 1
mode = "live_copy"
source = "source"
destination = "destination"

[migrations.tenant_copy.scope]
key_type = "hash"
matcher = { kind = "prefix", encoding = "text", value = "tenant:" }

[migrations.tenant_copy.policy]
conflict = "stop"
expiration = "keep_expiry_time"
verification = "quick"
allow_compatibility_skips = false

[migrations.tenant_copy.execution]
max_keys_per_second = 5000
timeout = "2h"

[migrations.tenant_copy.output]
progress = "auto"
progress_interval = "10s"
problems_file = "artifacts/tenant-copy.problems.jsonl"

varc migrate plan tenant_copy
varc migrate run tenant_copy

The same named connection can be the source and destination only when the selected databases differ. Cluster databases must be 0.

Export a portable bundle

Bundle export reads from one source and writes a .varcmigrate file:

[migrations.tenant_export]
schema_version = 1
mode = "bundle_export"
source = "source"
bundle_path = "artifacts/tenant.varcmigrate"
overwrite = "refuse"
estimated_uncompressed_bytes = 10737418240
minimum_free_after_export_bytes = 1073741824
source_labels = ["production", "pre-cutover"]

[migrations.tenant_export.scope]
matcher = { kind = "glob", encoding = "text", value = "tenant:*" }

[migrations.tenant_export.safety]
raw_export_acknowledged = true

[migrations.tenant_export.output]
progress = "plain"
problems_file = "artifacts/tenant-export.problems.jsonl"

varc migrate plan tenant_export
varc migrate run tenant_export

raw_export_acknowledged confirms that the portable bundle contains raw data. The free-space reserve is checked in addition to the conservative uncompressed size estimate. Existing bundles are refused unless overwrite is explicitly set to replace_validated_regular_file.

Import with a checkpoint

Bundle import validates the bundle before writing to the destination:

[migrations.tenant_import]
schema_version = 1
mode = "bundle_import"
bundle_path = "artifacts/tenant.varcmigrate"
destination = "destination"

[migrations.tenant_import.policy]
conflict = "stop"
expiration = "keep_expiry_time"
verification = "quick"
allow_compatibility_skips = false

[migrations.tenant_import.execution]
max_keys_per_second = 5000
timeout = "4h"

[migrations.tenant_import.output]
progress = "auto"
progress_file = "artifacts/tenant-import.progress.jsonl"
problems_file = "artifacts/tenant-import.problems.jsonl"
checkpoint_file = "artifacts/tenant-import.checkpoint.json"

varc migrate plan tenant_import
varc migrate run tenant_import

The versioned checkpoint records completed bundle chunks for a later reviewed import attempt. It does not turn an interrupted operation into a success or authorize blind replay of an indeterminate write.

Scope and policy

A migration scope can select one key type and one matcher; when both are set, they use AND semantics. Matchers support binary-safe prefix, suffix, and glob values encoded as text, hex, or Base64, plus a UTF-8 regex:

matcher = { kind = "prefix", encoding = "text", value = "tenant:" }
matcher = { kind = "suffix", encoding = "hex", value = "00ff" }
matcher = { kind = "glob", encoding = "base64", value = "dGVuYW50Oio=" }
matcher = { kind = "regex", pattern = "^tenant:\\x00[0-9]+$" }

For live copy and import, policy controls conflicts, expiry handling, verification, and compatibility skips:

conflict = "stop", "skip", or "replace";
expiration = "keep_expiry_time" or "restart_captured_ttl"; and
verification = "quick" or "none".

Failed writes and verification failures are always incomplete. Conflict skips are successful only under conflict = "skip". Compatibility skips remain incomplete unless allow_compatibility_skips is explicitly enabled.

Progress and output

Visible progress is written to stderr; the final human or JSON result remains the only stdout output. progress = "auto" selects a one-line TTY display in an interactive terminal and plain periodic heartbeats in CI. You can also choose tty, plain, jsonl, or none.

varc --non-interactive --format json migrate run tenant_copy \
  --progress jsonl \
  --progress-file artifacts/tenant-copy.progress.jsonl

Named migrations accept operational overrides such as progress paths and intervals, timeout, output replacement, checkpoint path, and the named safety acknowledgements. They reject target, mode, scope, and policy flags so a reviewed definition cannot be silently redirected. Omit the migration name and use varc migrate --help for the equivalent ad-hoc definition flags.

Problems files are bounded and binary-safe and never contain values or DUMP payloads. Existing output files are replaced only when replacement is enabled; symlinks are refused.

Interpret the result safely

Exit code 0 means the migration completed under the selected policy. Code 5 means the aggregate result is incomplete or indeterminate. Do not blindly retry an indeterminate write: review the terminal result, problems file, and any import checkpoint before deciding how to continue.

Clock skew above the engine threshold must be acknowledged with clock_skew_acknowledged = true or --acknowledge-clock-skew. This gate applies to live copy and bundle import. Raw-data acknowledgement applies only to bundle export.