Testing

Before deploying a custom action to production, validate its behavior thoroughly using S4E's testing tools: the schema validator, sandbox mode, and dry-run execution.

Schema Validation

Use the validation endpoint to check your action definition for structural errors before registration.

curl -X POST "https://api.s4e.io/api/actions/validate" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d @action-definition.json

A successful response confirms the schema is valid:

{
  "valid": true,
  "warnings": []
}

If there are issues, the response includes detailed error messages:

{
  "valid": false,
  "errors": [
    {
      "path": "$.parameters[0].type",
      "message": "Invalid parameter type 'text'. Allowed: string, integer, boolean, enum, secret."
    }
  ]
}

Tip

Run validation in your CI pipeline to catch schema errors before they reach the platform.

Sandbox Mode

Sandbox mode provides an isolated environment where actions execute without affecting production assets or external systems.

Enabling Sandbox Mode

Pass "sandbox": true when triggering an action:

curl -X POST "https://api.s4e.io/api/actions/act-block-ip/trigger" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target": "a-1001",
    "parameters": {
      "ip_address": "203.0.113.50",
      "duration_hours": 24
    },
    "sandbox": true
  }'

Sandbox Behavior

In sandbox mode:

  • External HTTP calls are intercepted and logged but not executed.
  • Shell commands run inside a restricted container with no network access.
  • Database writes are rolled back after execution.
  • All inputs, outputs, and side effects are captured in the execution log.

The execution response includes a sandbox_report field:

{
  "execution_id": "exec-9912",
  "status": "completed",
  "sandbox": true,
  "sandbox_report": {
    "intercepted_calls": [
      {
        "type": "http",
        "method": "POST",
        "url": "https://firewall.example.com/api/rules",
        "body": {"action": "block", "ip": "203.0.113.50"}
      }
    ],
    "commands_executed": [],
    "output": {"blocked": true, "rule_id": "simulated-001"}
  }
}

Dry-Run Execution

Dry-run mode validates parameter resolution, condition evaluation, and execution flow without invoking the executor.

curl -X POST "https://api.s4e.io/api/actions/act-block-ip/trigger" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "target": "a-1001",
    "parameters": {
      "ip_address": "203.0.113.50"
    },
    "dry_run": true
  }'

Response:

{
  "execution_id": "exec-dry-0045",
  "status": "dry_run_completed",
  "resolved_parameters": {
    "ip_address": "203.0.113.50",
    "duration_hours": 24
  },
  "conditions_met": true,
  "executor": "opservant",
  "estimated_duration_seconds": 5
}

Note

Dry-run does not execute any logic. It only validates that the action can be triggered with the given parameters.

Test Harness

For local development, use the S4E Action Test Harness to run actions outside the platform.

Installation

pip install s4e-sdk

Writing a Test

from s4e_sdk.testing import ActionTestHarness

harness = ActionTestHarness("./block-ip-action.json")

result = harness.run(
    parameters={"ip_address": "203.0.113.50", "duration_hours": 24},
    mock_responses={
        "https://firewall.example.com/api/rules": {
            "status": 200,
            "body": {"rule_id": "fw-123", "status": "active"}
        }
    }
)

assert result.status == "completed"
assert result.output["blocked"] is True
print("Test passed.")

Mocking External Dependencies

The harness intercepts outbound HTTP requests and returns configured mock responses. This lets you test action logic without connecting to real services.

harness.add_mock("https://slack.example.com/api/chat.postMessage", {
    "status": 200,
    "body": {"ok": True, "ts": "1234567890.123456"}
})

Common Test Scenarios

Scenario How to Test
Valid execution Trigger with correct parameters, verify output
Missing parameter Omit a required field, expect validation error
Invalid parameter Pass wrong type, expect 400 response
Timeout Set a very short timeout in sandbox, verify handling
External service down Mock with 500 response, verify retry/error behavior
Condition not met Set conditions that evaluate to false, verify skip

Checking Execution Logs

After any test execution, inspect the logs for debugging:

curl "https://api.s4e.io/api/actions/executions/exec-9912/logs" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "logs": [
    {"timestamp": "2026-04-28T12:00:01Z", "level": "INFO", "message": "Action triggered in sandbox mode."},
    {"timestamp": "2026-04-28T12:00:02Z", "level": "INFO", "message": "Parameter validation passed."},
    {"timestamp": "2026-04-28T12:00:02Z", "level": "INFO", "message": "Executor invoked: block-ip-executor"},
    {"timestamp": "2026-04-28T12:00:03Z", "level": "INFO", "message": "HTTP call intercepted: POST https://firewall.example.com/api/rules"},
    {"timestamp": "2026-04-28T12:00:03Z", "level": "INFO", "message": "Action completed successfully."}
  ]
}

CI/CD Integration

Add action validation and testing to your deployment pipeline:

# .github/workflows/test-actions.yml
name: Test Actions
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install s4e-sdk
      - run: |
          python -m s4e_sdk.actions.validate ./actions/*.json
      - run: |
          python -m pytest tests/actions/ -v

Warning

Never use production API keys in CI/CD pipelines. Create a dedicated testing key with limited scope.

Next Steps