Skip to main content

Organizing Specifications

Optimus supports two ways to deploy specifications

  • REST/GRPC
  • Optimus CLI deploy command

When using Optimus CLI to deploy, either manually or from a CI pipeline, it is advised to use a version control system like git. Here is a simple directory structure that can be used as a template for jobs and datastore resources, assuming there are 2 namespaces in a project.

.
├── optimus.yaml
├── preset.yaml
├── README.md
├── namespace-1
│   ├── jobs
|   │   ├── job1
|   │   ├── job2
|   │   └── this.yaml
│   └── resources
|       ├── bigquery
│       │   ├── table1
│       │   ├── table2
|       |   └── this.yaml
│       └── postgres
│           └── table1
├── namespace-2
├── jobs
└── resources

You might have also noticed there are this.yaml files being used in some directories. This file is used to share a single set of configurations across multiple sub-directories. For example, if you create a file at /namespace-1/jobs/this.yaml, then all subdirectories inside /namespaces-1/jobs will inherit this config as defaults. If the same config is specified in subdirectory, then subdirectory will override the parent defaults.

For example a this.yaml in /namespace-1/jobs

version: 1
schedule:
  interval: @daily
task:
  name: bq2bq
  config:
    BQ_SERVICE_ACCOUNT: "{{.secret.BQ_SERVICE_ACCOUNT}}"
behavior:
  depends_on_past: false
  retry:
    count: 1
    delay: 5s

and a job.yaml in /namespace-1/jobs/job1

name: sample_replace
owner: optimus@example.io
schedule:
  start_date: "2020-09-25"
  interval: 0 10 * * *
behavior:
  depends_on_past: true
task:
  name: bq2bq
  config:
    project: project_name
    dataset: project_dataset
    table: sample_replace
    load_method: REPLACE
  window:
    size: 48h
    offset: 24h

will result in final computed job.yaml during deployment as

version: 1
name: sample_replace
owner: optimus@example.io
schedule:
  start_date: "2020-10-06"
  interval: 0 10 * * *
behavior:
  depends_on_past: true
  retry:
    count: 1
    delay: 5s
task:
  name: bq2bq
  config:
    project: project_name
    dataset: project_dataset
    table: sample_replace
    load_method: REPLACE
    BQ_SERVICE_ACCOUNT: "{{.secret.BQ_SERVICE_ACCOUNT}}"
  window:
    size: 48h
    offset: 24h

Preset (since v0.10.0)

Window preset allows for easier usage of window configuration and can be specified through a YAML file. It is optional in nature and is recommended to be put in the same directory as optimus.yaml if being set. For more information on how to utilize window preset, please check this page.