Skip to content

Scheduled ECS Tasks Module

This module creates scheduled ECS tasks in AWS using EventBridge, AWS Step Functions, and standard ECS task definitions. The tasks are run in ECS Fargate.

It supports running a single task or multiple tasks in a sequence. It also supports retrying tasks if they fail.

Usage

module "scheduled_tasks" {
  source = "path_to_your_module"

  name_prefix      = "example-tasks"
  env              = "production"
  ecs_cluster_name = "example-cluster"
  subnet_ids       = ["subnet-12345678", "subnet-87654321"]
  security_groups  = ["sg-12345678"]

  tasks = [
    {
      name             = "example-task"
      container_image  = "123456789012.dkr.ecr.us-west-2.amazonaws.com/example-service:latest"
      cpu_architecture = "X86_64"
      command          = ["python", "app/task.py", "run"]
    },
    {
      name             = "example-task2"
      container_image  = "123456789012.dkr.ecr.us-west-2.amazonaws.com/example-service:latest"
      cpu_architecture = "X86_64"
      command          = ["python", "app/task.py", "run"]
      retry_task = {
        attempts = 3
      }
    }
  ]

  schedule_description = "Run tasks every hour"
  schedule_expression  = "cron(0 * * * ? *)"
}

Overriding the command and environment per execution

A task normally runs the command from its task definition, and the input passed to StartExecution is ignored. Set input_overrides = true on a task to let that input replace the container's command and environment instead:

{
  "command": ["node", "config/migrate-models.js", "--migration", "0001-example"],
  "environment": [{ "Name": "LOG_LEVEL", "Value": "debug" }]
}

Both keys are optional: command falls back to the task's own command and environment to no overrides. A scheduled execution therefore runs the task exactly as it would without the flag, because the EventBridge event it is started with carries neither key. Tasks without the flag ignore the input as before, so one state machine can mix scheduled and parameterised tasks.

The input is read by every flagged task in the execution — there is no per-task addressing. environment entries are Name/Value pairs as the ECS RunTask API spells them, and override same-named variables from the task definition.

ECS caps the whole overrides structure at 8,192 characters, punctuation included. Pass identifiers through the input and keep bulk configuration in S3 or Parameter Store.

Tests

tests/ holds terraform test files. The AWS provider is mocked, so they need no credentials and create nothing:

terraform init
terraform test

Modules

Name Source Version
eventbridge_scheduler terraform-aws-modules/eventbridge/aws 3.14.3
scheduled_task_failure_topic terraform-aws-modules/sns/aws 6.1.2
scheduled_task_success_topic terraform-aws-modules/sns/aws 6.1.2
step_function terraform-aws-modules/step-functions/aws 4.2.1
task_definitions ./modules/ecs_task n/a

Inputs

Name Description Type Default Required
create Whether to create the scheduled ECS tasks bool true no
create_schedule Whether to create the schedule bool true no
dd_api_key_parameter_name The name of the Parameter Store parameter containing the Datadog API key string "/datadog/DD_API_KEY" no
dd_apm_enabled Whether to enable Datadog APM bool true no
dd_apm_ignore_resources_string Datadog APM ignore resources string, e.g. 'GET /healthcheck string "" no
dd_enabled Whether to enable Datadog monitoring bool false no
dd_forwarder_function_arn The ARN of the Datadog forwarder function string null no
dd_log_level The log level for the Datadog agent string "INFO" no
dd_site The Datadog site to send data to string "datadoghq.eu" no
dd_tags Map of tags to apply to the Datadog agent map(string) {} no
ecs_cluster_name The name of the ECS cluster string n/a yes
env The environment the service is running in string n/a yes
ephemeral_storage The amount of ephemeral storage to allocate for the task. This parameter is used to expand the total amount of ephemeral storage available, beyond the default amount, for tasks hosted on AWS Fargate 
object({
    size_in_gib = number
  })
 null no
name_prefix The name prefix for the created resources string n/a yes
policy_jsons A list of JSON strings representing IAM policies to attach to the step function role list(string) [] no
schedule_description The description of the schedule in nice human readable format. E.g. 'Every day at 15:00' string null no
schedule_expression The expression to schedule the tasks. E.g. 'cron(0 15 * * ? *)' or 'rate(5 minutes)' string null no
security_groups The security groups to attach to the ECS service list(string) n/a yes
step_function_timeout The timeout for the step function number 86400 no
subnet_ids The subnets to place the ECS service list(string) n/a yes
tasks A list of tasks to schedule. Set input_overrides on a task to let the state machine execution input override that task's command and environment; see the README for the input contract 
list(object({
    name                         = string
    command                      = list(string)
    input_overrides              = optional(bool, false)
    container_image              = string
    container_cpu                = optional(number)
    container_memory             = optional(number)
    cpu_architecture             = optional(string)
    secretsmanager_secret_names  = optional(map(string))
    parameter_store_secret_names = optional(map(string))
    environment_variables        = optional(map(string))
    restart_policy = optional(object({
      enabled              = optional(bool)
      ignoredExitCodes     = optional(list(number))
      restartAttemptPeriod = optional(number)
    }))
    mount_points = optional(list(object({
      source_volume  = string
      container_path = string
      read_only      = bool
    })))
    retry_task = optional(object({
      attempts         = number
      backoff_rate     = optional(number)
      interval_seconds = optional(number)
    }))
    task_policy_json = optional(string)
  }))
 n/a yes

Outputs

Name Description
state_machine_arn The ARN of the Step Function state machine created by the module
state_machine_name The name of the Step Function state machine created by XXX module
step_function_definition The rendered Amazon States Language definition of the state machine
tasks_execution_iam_role_arns Map of task name to the ARN of the ECS task execution role created for that task. Use this to attach policies the execution role needs when starting a task, e.g. pulling an image through an ECR pull-through cache
tasks_execution_iam_role_names Map of task name to the name of the ECS task execution role created for that task. Use this to attach policies the execution role needs when starting a task, e.g. pulling an image through an ECR pull-through cache
tasks_iam_role_arns Map of task name to the ARN of the ECS task role created for that task. Use this to attach policies the running container needs, e.g. access to an S3 bucket
tasks_iam_role_names Map of task name to the name of the ECS task role created for that task. Use this to attach policies the running container needs, e.g. access to an S3 bucket