Deploy Lock

This is a tool to lock a cluster or service, in order to prevent people from deploying changes during test automation or restarting pods during an infrastructure incident.

Features

• in-memory data store, mostly for testing
• DynamoDB data store
• lock paths and recursive checking
• infer lock data from CI variables

Contents

• Deploy Lock
  • Features
  • Contents
  • Concepts
    • Lock Path
    • Lock Data
  • Usage
    • Command-line Interface
      • Basic Options
      • Lock Data Options
      • Storage Backend Options
      • Admission Controller Options
    • REST API
      • Endpoints

Concepts

Lock Path

Briefly describe paths.

More details here.

Lock Data

Each lock must contain the following fields:

interface Lock {
  type: 'automation' | 'deploy' | 'freeze' | 'incident' | 'maintenance';
  path: string;
  author: string;
  links: Map<string, string>;
  // often duplicates of path, but useful for cross-project locks
  source: string;

  // Timestamps, calculated from --duration and --until
  created_at: number;
  updated_at: number;
  expires_at: number;

  // CI fields, optional
  ci?: {
    project: string;
    ref: string;
    commit: string;
    pipeline: string;
    job: string;
  }
}

If $CI is not set, the ci sub-struct will not be present.

Usage

Command-line Interface

> deploy-lock check apps/staging/a/auth-app   # is equivalent to
> deploy-lock check apps && deploy-lock check apps/staging && deploy-lock check apps/staging/a && deploy-lock check apps/staging/a/auth-app
> deploy-lock check apps/staging/a/auth-app --recursive=false   # only checks the leaf node

> deploy-lock list
> deploy-lock list apps/staging    # list all locks within the apps/staging path

> deploy-lock lock apps/staging --type automation --duration 60m
> deploy-lock lock apps/staging/a/auth-app --type deploy --duration 5m
> deploy-lock lock apps/staging --until 2022-12-31T12:00   # local TZ, unless Z specified

> deploy-lock prune   # prune all expired locks
> deploy-lock prune apps/staging   # prune expired locks within the path
> deploy-lock prune apps/staging --now future-date   # prune locks that will expire by --now

> deploy-lock unlock apps/staging --type automation    # unlock type must match lock type

Basic Options

• command
  • one of check, list, lock, prune, unlock
• <path>
  • positional, required
  • string
  • lock path
  • always lowercase (forced in code)
  • /^[-a-z\/]+$/
• --now
  • number, optional
  • defaults to current epoch time
• --recursive
  • boolean
  • recursively check locks
  • defaults to true for check
  • defaults to false for lock, unlock
• --type
  • string, enum
  • type of lock
  • one of automation, deploy, freeze, incident, or maintenance

Lock Data Options

• --allow
  • array, strings
  • check types to allow while this lock is active
  • this does not allow creating other locks at the same path, it only changes whether the check returns allowed
• --author
  • string
  • defaults to $GITLAB_USER_EMAIL if $GITLAB_CI is set
  • defaults to $USER otherwise
• --duration
  • string
  • duration of lock, relative to now
  • may be given in epoch seconds (\d+), as an ISO-8601 date, or a human interval (30m)
  • mutually exclusive with --until
• --link
  • array, strings
• --source
  • string
  • each component:
    • first
      • defaults to $CLUSTER_NAME if set
      • defaults to path.split.0 otherwise
    • second
      • defaults to $DEPLOY_ENV if set
      • defaults to path.split.1 otherwise
    • third
      • defaults to $DEPLOY_TARGET if set
      • defaults to path.split.2 otherwise
    • fourth
      • defaults to --ci-project if set
      • defaults to $CI_PROJECT_PATH if set
      • defaults to path.split.3 otherwise
    • fifth
      • defaults to --ci-ref if set
      • defaults to $CI_COMMIT_REF_SLUG if set
      • defaults to path.split.4 otherwise
• --until
  • string, timestamp
  • duration of lock, absolute
  • may be given in epoch seconds (\d+) or as an ISO-8601 date (intervals are not allowed)
  • mutually exclusive with --duration
• --ci-project
  • optional string
  • project path
  • defaults to $CI_PROJECT_PATH if set
  • defaults to path.split.3 otherwise
• --ci-ref
  • optional string
  • branch or tag
  • defaults to $CI_COMMIT_REF_SLUG if set
  • defaults to path.split.4 otherwise
• --ci-commit
  • optional string
  • SHA of ref
  • defaults to $CI_COMMIT_SHA if set
• --ci-pipeline
  • optional string
  • pipeline ID
  • defaults to $CI_PIPELINE_ID if set
• --ci-job
  • optional string
  • job ID
  • defaults to $CI_JOB_ID if set

Storage Backend Options

• --storage
  • string
  • one of dynamodb, memory
• --region
  • string, optional
  • DynamoDB region name
• --table
  • string
  • DynamoDB table name
• --endpoint
  • string, optional
  • DynamoDB endpoint
  • set to http://localhost:8000 for testing with DynamoDB local
• --fake
  • string, optional
  • a fake lock that should be added to the in-memory data store
  • the in-memory data store always starts empty, this is the only way to have an existing lock

Admission Controller Options

• --admission-base
  • string
  • base path for admission controller requests
  • usually the path to the cluster within the deploy path, like aws/staging/us-east-1/apps/a
• --admission-path
  • array, strings
  • path keys to use when checking admission controller requests
  • must be fields within the AdmissionReview kind
  • usually something like [namespace, name] or [namespace, requestResource.group, requestResource.resource, name]

REST API

Endpoints

• /admission POST
  • Kubernetes admission controller webhook
• /locks GET
  • equivalent to deploy-lock list
• /locks POST
  • equivalent to deploy-lock lock
  • path taken from request body
• /locks DELETE
  • equivalent to deploy-lock prune
• /locks/:path GET
  • equivalent to deploy-lock check
• /locks/:path PUT
  • equivalent to deploy-lock lock
  • path taken from URL path
• /locks/:path DELETE
  • equivalent to deploy-lock unlock
• /ok
  • health check