feat(cli): improve the "up" commands --from and --to options

The given values can either be migration names or relative paths to migration files.
The given migration must exist to avoid accidentally running migrations that wasn't intended to run.
This commit is contained in:
Joakim Carlstein 2024-01-25 17:59:20 +01:00 committed by Joakim Carlstein
parent 986456b038
commit 9109238b86
12 changed files with 346 additions and 65 deletions

View file

@ -0,0 +1,5 @@
---
'@emigrate/cli': minor
---
When the `--from` or `--to` CLI options are used the given migration name (or path to migration file) must exist. This is a BREAKING CHANGE from before. The reasoning is that by forcing the migrations to exist you avoid accidentally running migrations you don't intend to, because a simple typo could have the effect that many unwanted migrations is executed so it's better to show an error if that's the case.

View file

@ -0,0 +1,5 @@
---
'@emigrate/docs': minor
---
Document the changes to the "up" command's `--from` and `--to` options, specifically that they can take relative file paths and that the given migration must exist.

View file

@ -0,0 +1,5 @@
---
'@emigrate/cli': minor
---
Add support for passing relative paths to migration files as the `--from` and `--to` CLI options. This is very useful from terminals that support autocomplete for file paths. It also makes it possible to copy the path to a migration file from Emigrate's output and use that as either `--from` and `--to` directly.

View file

@ -49,22 +49,35 @@ Run all pending migrations
Options:
-h, --help Show this help message and exit
-d, --directory <path> The directory where the migration files are located (required)
-i, --import <module> Additional modules/packages to import before running the migrations (can be specified multiple times)
For example if you want to use Dotenv to load environment variables or when using TypeScript
-s, --storage <name> The storage to use for where to store the migration history (required)
-p, --plugin <name> The plugin(s) to use (can be specified multiple times)
-r, --reporter <name> The reporter to use for reporting the migration progress
-l, --limit <count> Limit the number of migrations to run
-f, --from <name> Start running migrations from the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-t, --to <name> Skip migrations after the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-f, --from <name/path> Start running migrations from the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those after it lexicographically will be run
-t, --to <name/path> Skip migrations after the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those before it lexicographically will be run
--dry List the pending migrations that would be run without actually running them
--color Force color output (this option is passed to the reporter)
--no-color Disable color output (this option is passed to the reporter)
--no-execution Mark the migrations as executed and successful without actually running them,
which is useful if you want to mark migrations as successful after running them manually
--abort-respite <sec> The number of seconds to wait before abandoning running migrations after the command has been aborted (default: 10)
Examples:

View file

@ -75,20 +75,34 @@ and "skipped" for the migrations that also haven't been run but won't because of
The directory where the migration files are located. The given path should be absolute or relative to the current working directory.
### `-f`, `--from <name>`
### `-f`, `--from <name/path>`
The name of the migration to start from. This can be used to run only a subset of the pending migrations.
The given migration name does not need to exist and is compared in lexicographical order with the migration names, so it can be a prefix of a migration name or similar.
The given migration need to exist and is compared in lexicographical order with all migrations, the migration with the same name and those lexicographically after it will be migrated.
It's okay to use an already executed migration as the "from" migration, it won't be executed again.
The reason for why the given migration name must exist and cannot be just a prefix is to avoid accidentally running migrations that you didn't intend to run.
The given name can also be a relative path to a migration file, which makes it easier to use with terminals that support tab completion
or when copying the output from Emigrate and using it directly as the value of the `--from` option.
Relative paths are resolved relative to the current working directory.
Can be combined with `--dry` which will show "pending" for the migrations that would be run if not in dry-run mode,
and "skipped" for the migrations that also haven't been run but won't because of the set "from".
### `-t`, `--to <name>`
### `-t`, `--to <name/path>`
The name of the migration to end at. This can be used to run only a subset of the pending migrations.
The given migration name does not need to exist and is compared in lexicographical order with the migration names, so it can be a prefix of a migration name or similar.
The given migration name need to exist and is compared in lexicographical order with all migrations, the migration with the same name and those lexicographically before it will be migrated.
It's okay to use an already executed migration as the "to" migration, it won't be executed again.
The reason for why the given migration name must exist and cannot be just a prefix is to avoid accidentally running migrations that you didn't intend to run.
The given name can also be a relative path to a migration file, which makes it easier to use with terminals that support tab completion
or when copying the output from Emigrate and using it directly as the value of the `--to` option.
Relative paths are resolved relative to the current working directory.
Can be combined with `--dry` which will show "pending" for the migrations that would be run if not in dry-run mode,
and "skipped" for the migrations that also haven't been run but won't because of the set "to".

View file

@ -28,22 +28,35 @@ Run all pending migrations
Options:
-h, --help Show this help message and exit
-d, --directory <path> The directory where the migration files are located (required)
-i, --import <module> Additional modules/packages to import before running the migrations (can be specified multiple times)
For example if you want to use Dotenv to load environment variables or when using TypeScript
-s, --storage <name> The storage to use for where to store the migration history (required)
-p, --plugin <name> The plugin(s) to use (can be specified multiple times)
-r, --reporter <name> The reporter to use for reporting the migration progress
-l, --limit <count> Limit the number of migrations to run
-f, --from <name> Start running migrations from the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-t, --to <name> Skip migrations after the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-f, --from <name/path> Start running migrations from the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those after it lexicographically will be run
-t, --to <name/path> Skip migrations after the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those before it lexicographically will be run
--dry List the pending migrations that would be run without actually running them
--color Force color output (this option is passed to the reporter)
--no-color Disable color output (this option is passed to the reporter)
--no-execution Mark the migrations as executed and successful without actually running them,
which is useful if you want to mark migrations as successful after running them manually
--abort-respite <sec> The number of seconds to wait before abandoning running migrations after the command has been aborted (default: 10)
Examples:

View file

@ -93,22 +93,35 @@ Run all pending migrations
Options:
-h, --help Show this help message and exit
-d, --directory <path> The directory where the migration files are located (required)
-i, --import <module> Additional modules/packages to import before running the migrations (can be specified multiple times)
For example if you want to use Dotenv to load environment variables or when using TypeScript
-s, --storage <name> The storage to use for where to store the migration history (required)
-p, --plugin <name> The plugin(s) to use (can be specified multiple times)
-r, --reporter <name> The reporter to use for reporting the migration progress
-l, --limit <count> Limit the number of migrations to run
-f, --from <name> Start running migrations from the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-t, --to <name> Skip migrations after the given migration name, the given name doesn't need to exist
and is compared in lexicographical order
-f, --from <name/path> Start running migrations from the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those after it lexicographically will be run
-t, --to <name/path> Skip migrations after the given migration name or relative file path to a migration file,
the given name or path needs to exist. The same migration and those before it lexicographically will be run
--dry List the pending migrations that would be run without actually running them
--color Force color output (this option is passed to the reporter)
--no-color Disable color output (this option is passed to the reporter)
--no-execution Mark the migrations as executed and successful without actually running them,
which is useful if you want to mark migrations as successful after running them manually
--abort-respite <sec> The number of seconds to wait before abandoning running migrations after the command has been aborted (default: ${DEFAULT_RESPITE_SECONDS})
Examples:

View file

@ -3,7 +3,6 @@ import { BadOptionError, MissingOptionError, StorageInitError, toError } from '.
import { type Config } from '../types.js';
import { exec } from '../exec.js';
import { migrationRunner } from '../migration-runner.js';
import { arrayFromAsync } from '../array-from-async.js';
import { collectMigrations } from '../collect-migrations.js';
import { version } from '../get-package-info.js';
@ -56,7 +55,7 @@ export default async function listCommand({
dry: true,
reporter,
storage,
migrations: await arrayFromAsync(collectedMigrations),
migrations: collectedMigrations,
async validate() {
// No-op
},

View file

@ -278,7 +278,100 @@ describe('up', () => {
]);
});
it('returns 0 and finishes without an error when pending migrations after given "from" parameter are run successfully, even when the "from" is not an existing migration', async () => {
it('returns 0 and finishes without an error when pending migrations after given "from" parameter are run successfully', async () => {
const migration = mock.fn(async () => {
// Success
});
const { reporter, run } = getUpCommand(
[
'1_some_already_run_migration.js',
'2_some_migration.js',
'3_existing_migration.js',
'4_some_other_migration.js',
],
getStorage(['1_some_already_run_migration.js']),
[
{
loadableExtensions: ['.js'],
async loadMigration() {
return migration;
},
},
],
);
const exitCode = await run({ from: '3_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '3_existing_migration.js', status: 'done', started: true },
{ name: '4_some_other_migration.js', status: 'done', started: true },
]);
assert.strictEqual(migration.mock.calls.length, 2);
});
it('returns 0 and finishes without an error when pending migrations after given "from" parameter are run successfully, when the "from" parameter is a relative path', async () => {
const migration = mock.fn(async () => {
// Success
});
const { reporter, run } = getUpCommand(
[
'1_some_already_run_migration.js',
'2_some_migration.js',
'3_existing_migration.js',
'4_some_other_migration.js',
],
getStorage(['1_some_already_run_migration.js']),
[
{
loadableExtensions: ['.js'],
async loadMigration() {
return migration;
},
},
],
);
const exitCode = await run({ from: 'migrations/3_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '3_existing_migration.js', status: 'done', started: true },
{ name: '4_some_other_migration.js', status: 'done', started: true },
]);
assert.strictEqual(migration.mock.calls.length, 2);
});
it('returns 0 and runs all pending migrations, if "from" is an already executed migration', async () => {
const migration = mock.fn(async () => {
// Success
});
const { reporter, run } = getUpCommand(
['1_some_already_run_migration.js', '2_some_migration.js', '4_some_other_migration.js'],
getStorage(['1_some_already_run_migration.js']),
[
{
loadableExtensions: ['.js'],
async loadMigration() {
return migration;
},
},
],
);
const exitCode = await run({ from: '1_some_already_run_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'done', started: true },
{ name: '4_some_other_migration.js', status: 'done', started: true },
]);
assert.strictEqual(migration.mock.calls.length, 2);
});
it('returns 1 and finishes with an error when the given "from" migration name does not exist', async () => {
const migration = mock.fn(async () => {
// Success
});
@ -297,30 +390,71 @@ describe('up', () => {
const exitCode = await run({ from: '3_non_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
assert.strictEqual(exitCode, 1, 'Exit code');
assertPreconditionsFulfilled(
{ dry: false },
reporter,
[
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '4_some_other_migration.js', status: 'done', started: true },
]);
assert.strictEqual(migration.mock.calls.length, 1);
{ name: '4_some_other_migration.js', status: 'skipped' },
],
BadOptionError.fromOption(
'from',
'The "from" migration: "migrations/3_non_existing_migration.js" was not found',
),
);
assert.strictEqual(migration.mock.calls.length, 0);
});
it('returns 0 and finishes without an error when pending migrations after given "from" parameter are validated and listed successfully in dry-mode, even when the "from" is not an existing migration', async () => {
it('returns 0 and finishes without an error when pending migrations after given "from" parameter are validated and listed successfully in dry-mode', async () => {
const { reporter, run } = getUpCommand(
['1_some_already_run_migration.js', '2_some_migration.js', '4_some_other_migration.js'],
['1_some_already_run_migration.js', '2_some_migration.js', '3_some_other_migration.js'],
getStorage(['1_some_already_run_migration.js']),
);
const exitCode = await run({ dry: true, from: '3_non_existing_migration.js' });
const exitCode = await run({ dry: true, from: '3_some_other_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: true }, reporter, [
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '4_some_other_migration.js', status: 'pending' },
{ name: '3_some_other_migration.js', status: 'pending' },
]);
});
it('returns 0 and finishes without an error when pending migrations before given "to" parameter are run successfully, even when the "to" is not an existing migration', async () => {
it('returns 0 and finishes without an error when pending migrations before given "to" parameter are run successfully', async () => {
const migration = mock.fn(async () => {
// Success
});
const { reporter, run } = getUpCommand(
[
'1_some_already_run_migration.js',
'2_some_migration.js',
'3_existing_migration.js',
'4_some_other_migration.js',
],
getStorage(['1_some_already_run_migration.js']),
[
{
loadableExtensions: ['.js'],
async loadMigration() {
return migration;
},
},
],
);
const exitCode = await run({ to: '3_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'done', started: true },
{ name: '3_existing_migration.js', status: 'done', started: true },
{ name: '4_some_other_migration.js', status: 'skipped' },
]);
assert.strictEqual(migration.mock.calls.length, 2);
});
it('returns 1 and finishes with an error when the given "to" migration name does not exist', async () => {
const migration = mock.fn(async () => {
// Success
});
@ -339,25 +473,63 @@ describe('up', () => {
const exitCode = await run({ to: '3_non_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'done', started: true },
assert.strictEqual(exitCode, 1, 'Exit code');
assertPreconditionsFulfilled(
{ dry: false },
reporter,
[
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '4_some_other_migration.js', status: 'skipped' },
]);
assert.strictEqual(migration.mock.calls.length, 1);
],
BadOptionError.fromOption('to', 'The "to" migration: "migrations/3_non_existing_migration.js" was not found'),
);
assert.strictEqual(migration.mock.calls.length, 0);
});
it('returns 0 and finishes without an error when pending migrations after given "to" parameter are validated and listed successfully in dry-mode, even when the "to" is not an existing migration', async () => {
it('returns 0 and runs no migrations, if "to" is an already executed migration', async () => {
const migration = mock.fn(async () => {
// Success
});
const { reporter, run } = getUpCommand(
['1_some_already_run_migration.js', '2_some_migration.js', '4_some_other_migration.js'],
getStorage(['1_some_already_run_migration.js']),
[
{
loadableExtensions: ['.js'],
async loadMigration() {
return migration;
},
},
],
);
const exitCode = await run({ dry: true, to: '3_non_existing_migration.js' });
const exitCode = await run({ to: '1_some_already_run_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: false }, reporter, [
{ name: '2_some_migration.js', status: 'skipped' },
{ name: '4_some_other_migration.js', status: 'skipped' },
]);
assert.strictEqual(migration.mock.calls.length, 0);
});
it('returns 0 and finishes without an error when pending migrations after given "to" parameter are validated and listed successfully in dry-mode', async () => {
const { reporter, run } = getUpCommand(
[
'1_some_already_run_migration.js',
'2_some_migration.js',
'3_existing_migration.js',
'4_some_other_migration.js',
],
getStorage(['1_some_already_run_migration.js']),
);
const exitCode = await run({ dry: true, to: '3_existing_migration.js' });
assert.strictEqual(exitCode, 0, 'Exit code');
assertPreconditionsFulfilled({ dry: true }, reporter, [
{ name: '2_some_migration.js', status: 'pending' },
{ name: '3_existing_migration.js', status: 'pending' },
{ name: '4_some_other_migration.js', status: 'skipped' },
]);
});

View file

@ -1,3 +1,4 @@
import path from 'node:path';
import { getOrLoadPlugins, getOrLoadReporter, getOrLoadStorage } from '@emigrate/plugin-tools';
import { isFinishedMigration, type LoaderPlugin } from '@emigrate/types';
import { BadOptionError, MigrationLoadError, MissingOptionError, StorageInitError, toError } from '../errors.js';
@ -6,9 +7,7 @@ import { withLeadingPeriod } from '../with-leading-period.js';
import { type GetMigrationsFunction } from '../get-migrations.js';
import { exec } from '../exec.js';
import { migrationRunner } from '../migration-runner.js';
import { filterAsync } from '../filter-async.js';
import { collectMigrations } from '../collect-migrations.js';
import { arrayFromAsync } from '../array-from-async.js';
import { version } from '../get-package-info.js';
type ExtraFlags = {
@ -72,10 +71,7 @@ export default async function upCommand({
}
try {
const collectedMigrations = filterAsync(
collectMigrations(cwd, directory, storage.getHistory(), getMigrations),
(migration) => !isFinishedMigration(migration) || migration.status === 'failed',
);
const collectedMigrations = collectMigrations(cwd, directory, storage.getHistory(), getMigrations);
const loaderPlugins = await getOrLoadPlugins('loader', [lazyPluginLoaderJs, ...plugins]);
@ -93,6 +89,14 @@ export default async function upCommand({
return loaderByExtension.get(extension);
};
if (from && !from.includes(path.sep)) {
from = path.join(directory, from);
}
if (to && !to.includes(path.sep)) {
to = path.join(directory, to);
}
const error = await migrationRunner({
dry,
limit,
@ -102,7 +106,10 @@ export default async function upCommand({
abortRespite,
reporter,
storage,
migrations: await arrayFromAsync(collectedMigrations),
migrations: collectedMigrations,
migrationFilter(migration) {
return !isFinishedMigration(migration) || migration.status === 'failed';
},
async validate(migration) {
if (noExecution) {
return;

View file

@ -1,13 +0,0 @@
export function filterAsync<T, S extends T>(
iterable: AsyncIterable<T>,
filter: (item: T) => item is S,
): AsyncIterable<S>;
export function filterAsync<T>(iterable: AsyncIterable<T>, filter: (item: T) => unknown): AsyncIterable<T>;
export async function* filterAsync<T>(iterable: AsyncIterable<T>, filter: (item: T) => unknown): AsyncIterable<T> {
for await (const item of iterable) {
if (filter(item)) {
yield item;
}
}
}

View file

@ -9,7 +9,7 @@ import {
type FailedMigrationMetadata,
type SuccessfulMigrationMetadata,
} from '@emigrate/types';
import { toError, EmigrateError, MigrationRunError, toSerializedError } from './errors.js';
import { toError, EmigrateError, MigrationRunError, toSerializedError, BadOptionError } from './errors.js';
import { exec } from './exec.js';
import { getDuration } from './get-duration.js';
@ -22,7 +22,8 @@ type MigrationRunnerParameters = {
abortRespite?: number;
reporter: EmigrateReporter;
storage: Storage;
migrations: Array<MigrationMetadata | MigrationMetadataFinished>;
migrations: AsyncIterable<MigrationMetadata | MigrationMetadataFinished>;
migrationFilter?: (migration: MigrationMetadata | MigrationMetadataFinished) => boolean;
validate: (migration: MigrationMetadata) => Promise<void>;
execute: (migration: MigrationMetadata) => Promise<void>;
};
@ -39,9 +40,9 @@ export const migrationRunner = async ({
migrations,
validate,
execute,
migrationFilter = () => true,
}: MigrationRunnerParameters): Promise<Error | undefined> => {
await reporter.onCollectedMigrations?.(migrations);
const collectedMigrations: Array<MigrationMetadata | MigrationMetadataFinished> = [];
const validatedMigrations: Array<MigrationMetadata | MigrationMetadataFinished> = [];
const migrationsToLock: MigrationMetadata[] = [];
@ -63,15 +64,32 @@ export const migrationRunner = async ({
{ once: true },
);
let fromFound = false;
let toFound = false;
for await (const migration of migrations) {
if (from && migration.relativeFilePath === from) {
fromFound = true;
}
if (to && migration.relativeFilePath === to) {
toFound = true;
}
if (!migrationFilter(migration)) {
continue;
}
collectedMigrations.push(migration);
if (isFinishedMigration(migration)) {
skip ||= migration.status === 'failed' || migration.status === 'skipped';
validatedMigrations.push(migration);
} else if (
skip ||
Boolean(from && migration.name < from) ||
Boolean(to && migration.name > to) ||
Boolean(from && migration.relativeFilePath < from) ||
Boolean(to && migration.relativeFilePath > to) ||
(limit && migrationsToLock.length >= limit)
) {
validatedMigrations.push({
@ -107,6 +125,32 @@ export const migrationRunner = async ({
}
}
await reporter.onCollectedMigrations?.(collectedMigrations);
let optionError: Error | undefined;
if (from && !fromFound) {
optionError = BadOptionError.fromOption('from', `The "from" migration: "${from}" was not found`);
} else if (to && !toFound) {
optionError = BadOptionError.fromOption('to', `The "to" migration: "${to}" was not found`);
}
if (optionError) {
dry = true;
skip = true;
for (const migration of migrationsToLock) {
const validatedIndex = validatedMigrations.indexOf(migration);
validatedMigrations[validatedIndex] = {
...migration,
status: 'skipped',
};
}
migrationsToLock.length = 0;
}
const [lockedMigrations, lockError] = dry
? [migrationsToLock]
: await exec(async () => storage.lock(migrationsToLock), { abortSignal, abortRespite });
@ -227,7 +271,11 @@ export const migrationRunner = async ({
? MigrationRunError.fromMetadata(firstFailed)
: undefined;
const error =
unlockError ?? firstError ?? lockError ?? (abortSignal?.aborted ? toError(abortSignal.reason) : undefined);
optionError ??
unlockError ??
firstError ??
lockError ??
(abortSignal?.aborted ? toError(abortSignal.reason) : undefined);
await reporter.onFinished?.(finishedMigrations, error);