Deploy Lock

This is a tool to lock a cluster or service, in order to prevent people from deploying changes during test automation or restarting pods during an infrastructure incident.

Features

in-memory data store, mostly for testing
DynamoDB data store
lock paths and recursive checking
infer lock data from CI variables

Deploy Lock

Concepts

Lock Path

Briefly describe paths.

More details here.

Lock Data

Each lock must contain the following fields:

interface Lock {
  type: 'automation' | 'deploy' | 'freeze' | 'incident' | 'maintenance';
  path: string;
  author: string;
  links: Map<string, string>;
  // often duplicates of path, but useful for cross-project locks
  source: string;

  // Timestamps, calculated from --duration and --until
  created_at: number;
  updated_at: number;
  expires_at: number;

  // CI fields, optional
  ci?: {
    project: string;
    ref: string;
    commit: string;
    pipeline: string;
    job: string;
  }
}

If $CI is not set, the ci sub-struct will not be present.

Usage

Command-line Interface

> deploy-lock check apps/staging/a/auth-app   # is equivalent to
> deploy-lock check apps && deploy-lock check apps/staging && deploy-lock check apps/staging/a && deploy-lock check apps/staging/a/auth-app
> deploy-lock check apps/staging/a/auth-app --recursive=false   # only checks the leaf node

> deploy-lock list
> deploy-lock list apps/staging    # list all locks within the apps/staging path

> deploy-lock lock apps/staging --type automation --duration 60m
> deploy-lock lock apps/staging/a/auth-app --type deploy --duration 5m
> deploy-lock lock apps/staging --until 2022-12-31T12:00   # local TZ, unless Z specified

> deploy-lock prune   # prune all expired locks
> deploy-lock prune apps/staging   # prune expired locks within the path
> deploy-lock prune apps/staging --now future-date   # prune locks that will expire by --now

> deploy-lock unlock apps/staging --type automation    # unlock type must match lock type

Basic Options

command
- one of check, list, lock, prune, unlock
<path>
- positional, required
- string
- lock path
- always lowercase (forced in code)
- /^[-a-z\/]+$/
--now
- number, optional
- defaults to current epoch time
--recursive
- boolean
- recursively check locks
- defaults to true for check
- defaults to false for lock, unlock
--type
- string, enum
- type of lock
- one of automation, deploy, freeze, incident, or maintenance

Lock Data Options

--allow
- array, strings
- check types to allow while this lock is active
- this does not allow creating other locks at the same path, it only changes whether the check returns allowed
--author
- string
- defaults to $GITLAB_USER_EMAIL if $GITLAB_CI is set
- defaults to $USER otherwise
--duration
- string
- duration of lock, relative to now
- may be given in epoch seconds (\d+), as an ISO-8601 date, or a human interval (30m)
- mutually exclusive with --until
--link
- array, strings
--source
- string
- each component:
  - first
    - defaults to $CLUSTER_NAME if set
    - defaults to path.split.0 otherwise
  - second
    - defaults to $DEPLOY_ENV if set
    - defaults to path.split.1 otherwise
  - third
    - defaults to $DEPLOY_TARGET if set
    - defaults to path.split.2 otherwise
  - fourth
    - defaults to --ci-project if set
    - defaults to $CI_PROJECT_PATH if set
    - defaults to path.split.3 otherwise
  - fifth
    - defaults to --ci-ref if set
    - defaults to $CI_COMMIT_REF_SLUG if set
    - defaults to path.split.4 otherwise
--until
- string, timestamp
- duration of lock, absolute
- may be given in epoch seconds (\d+) or as an ISO-8601 date (intervals are not allowed)
- mutually exclusive with --duration
--ci-project
- optional string
- project path
- defaults to $CI_PROJECT_PATH if set
- defaults to path.split.3 otherwise
--ci-ref
- optional string
- branch or tag
- defaults to $CI_COMMIT_REF_SLUG if set
- defaults to path.split.4 otherwise
--ci-commit
- optional string
- SHA of ref
- defaults to $CI_COMMIT_SHA if set
--ci-pipeline
- optional string
- pipeline ID
- defaults to $CI_PIPELINE_ID if set
--ci-job
- optional string
- job ID
- defaults to $CI_JOB_ID if set

Storage Backend Options

--storage
- string
- one of dynamodb, memory
--region
- string, optional
- DynamoDB region name
--table
- string
- DynamoDB table name
--endpoint
- string, optional
- DynamoDB endpoint
- set to http://localhost:8000 for testing with DynamoDB local
--fake
- string, optional
- a fake lock that should be added to the in-memory data store
- the in-memory data store always starts empty, this is the only way to have an existing lock

Admission Controller Options

--admission-base
- string
- base path for admission controller requests
- usually the path to the cluster within the deploy path, like aws/staging/us-east-1/apps/a
--admission-path
- array, strings
- path keys to use when checking admission controller requests
- must be fields within the AdmissionReview kind
- usually something like [namespace, name] or [namespace, requestResource.group, requestResource.resource, name]

REST API

Endpoints

/admission POST
- Kubernetes admission controller webhook
/locks GET
- equivalent to deploy-lock list
/locks POST
- equivalent to deploy-lock lock
- path taken from request body
/locks DELETE
- equivalent to deploy-lock prune
/locks/:path GET
- equivalent to deploy-lock check
/locks/:path PUT
- equivalent to deploy-lock lock
- path taken from URL path
/locks/:path DELETE
- equivalent to deploy-lock unlock
/ok
- health check

6.5 KiB Raw Blame History