Skip to main content

Audit log

The audit log keeps a record of important actions in Semaphore: who signed in, who changed a user or a role, who created an API token. Each event shows who did it, when, from which address, and whether it worked. Use it to find out what happened on your installation, or send the events to your SIEM to keep them next to the rest of your logs.

The audit log is available in every edition. Filtering, exporting it as a file and sending events to a SIEM require Semaphore Pro.

What is recorded​

Semaphore currently records sign-in, account, project and task activity:

  • sign-ins, failed sign-in attempts, sign-outs and two-factor checks;
  • rejected API tokens, denied requests and blocked cross-site requests;
  • changes to users, passwords, two-factor authentication, external identities and API tokens;
  • changes to project members, roles and template permissions;
  • changes to projects, inventories, repositories, templates, schedules, integrations, host configs, environments, credentials and secret storages, and project backup exports and restores;
  • changes to system settings and Pro license activation;
  • task starts with their trigger (API, schedule, integration, autorun, workflow), approvals, stops, completions and deleted task history;
  • runner changes, registrations (including refused registration tokens), unregistrations and runner reports with an invalid status;
  • exports of the audit log as a file;
  • every server start.

For the full list, see Audit events.

Passwords, tokens, secret values and task output never appear in audit events. API tokens are shown by a fingerprint instead of their value. A failed sign-in keeps the login name that was entered, so it can contain an email address.

In a task completion event, metadata.result shows the status Semaphore gave the task, and metadata.end_reason says why Semaphore ended it: timeout when it ran too long, runner_lost when its runner stopped responding. Task arguments, variables, runner tags and tokens are not recorded.

During a rolling upgrade of an HA cluster, a task started on an upgraded node and finished on a node that is not upgraded yet has no completion event.

Repository URLs, host config URLs and integration aliases are not recorded either.

Sometimes Semaphore saves or deletes an object, but a later part of the same request fails. The UI or API then shows an error, although the object was created or deleted. Such an event is recorded as a success with metadata.partial=true, and reason says what did not complete:

  • secret_failed: an environment was saved or deleted, but some of its secrets were not saved or not removed;
  • inventory_failed: a template was created, but its Terraform workspace inventory was not;
  • restore_failed: a project was restored from a backup, but not all of its objects were;
  • setup_failed: a project was created, but not fully set up, for example its creator was not added as owner.

Enable the audit log​

The audit log is off by default. To turn it on, set audit.enabled and give your installation a name in audit.instance_id:

{
  "audit": {
    "enabled": true,
    "instance_id": "prod-eu"
  }
}

Or using environment variables:

SEMAPHORE_AUDIT_ENABLED=true
SEMAPHORE_AUDIT_INSTANCE_ID=prod-eu

The instance ID is 1 to 255 characters without spaces. It is added to every event, so you can tell your installations apart when they send events to the same place.

Restart Semaphore. Recording starts after the restart; earlier actions are not added. For all options, see Configuration options.

Record the client address behind a proxy​

If Semaphore runs behind a reverse proxy, events show the address of the proxy instead of the user. To record the real client address, list your proxy networks in audit.trusted_proxy_cidrs:

{
  "audit": {
    "enabled": true,
    "instance_id": "prod-eu",
    "trusted_proxy_cidrs": ["10.0.0.0/8"]
  }
}

Or using environment variables:

SEMAPHORE_AUDIT_TRUSTED_PROXY_CIDRS='["10.0.0.0/8"]'

Semaphore then takes the client address from X-Forwarded-For or X-Real-IP, but only for requests that come from these networks. If requests pass through several proxies, list all of them. Do not list the networks your users connect from: anyone there could set these headers to any address.

Storage​

Events are stored in the Semaphore database, so your regular database backups include them. By default Semaphore keeps every event. To delete old events, set a retention period in days:

{
  "audit": {
    "retention_days": 365
  }
}

Or using an environment variable: SEMAPHORE_AUDIT_RETENTION_DAYS=365.

Semaphore deletes older events once an hour and records an audit.retention/delete event with the number of deleted events. If you export events to a SIEM, choose a period longer than the longest SIEM outage you want to survive: events that are older than the period are deleted even if they were not sent.

Retention also runs when Semaphore starts. In an HA installation, use the same retention_days on every node.

The audit log never gets in the way of your users. If an event cannot be saved, Semaphore writes an error to the server log and the action continues as usual.

Export to a SIEM Pro​

Semaphore Pro can send audit events to a Syslog receiver over TLS, such as rsyslog or Vector, and to any receiver of the Splunk HTTP Event Collector (HEC) protocol, such as Splunk, Vector, Fluent Bit, the OpenTelemetry Collector or Cribl. You can set up one Syslog and one HEC destination, or both at once.

You will need:

  • the receiver hostname and port;
  • a name for this destination, such as security-syslog;
  • the CA certificate of the receiver, if the Semaphore host does not trust it already.

Add audit.syslog to config.json:

{
  "audit": {
    "enabled": true,
    "instance_id": "prod-eu",
    "syslog": {
      "id": "security-syslog",
      "address": "siem.example.com:6514",
      "ca_file": "/etc/semaphore/siem-ca.pem",
      "server_name": "siem.example.com",
      "timeout": "10s"
    }
  }
}

Or using environment variables:

SEMAPHORE_AUDIT_SYSLOG_ID=security-syslog
SEMAPHORE_AUDIT_SYSLOG_ADDRESS=siem.example.com:6514
SEMAPHORE_AUDIT_SYSLOG_CA_FILE=/etc/semaphore/siem-ca.pem
SEMAPHORE_AUDIT_SYSLOG_SERVER_NAME=siem.example.com
SEMAPHORE_AUDIT_SYSLOG_TIMEOUT=10s

id and address are required. Semaphore remembers which events it has already sent to each destination, so keep the same id when you change the address or the certificate. A new id starts with new events.

Semaphore always checks the receiver certificate and uses TLS 1.2 or later. ca_file adds your CA to the trusted certificates, and server_name sets the name to check in the certificate when it differs from the address.

Restart Semaphore. If the settings are invalid or the CA file cannot be read, Semaphore does not start.

Check that events arrive​

Semaphore records an event every time it starts. After the restart, look for it at the receiver: event_code is audit.lifecycle, action is start, and metadata.destinations includes your destination ID.

Send events over HEC​

You will need the HEC endpoint URL, a HEC token, a name for this destination such as security-hec, and the CA certificate of the receiver if the Semaphore host does not trust it already.

{
  "audit": {
    "enabled": true,
    "instance_id": "prod-eu",
    "splunk_hec": {
      "id": "security-hec",
      "url": "https://splunk.example.com:8088/services/collector/event",
      "token": "<HEC token>",
      "index": "security",
      "ca_file": "/etc/semaphore/siem-ca.pem"
    }
  }
}

Or using environment variables:

SEMAPHORE_AUDIT_SPLUNK_HEC_ID=security-hec
SEMAPHORE_AUDIT_SPLUNK_HEC_URL=https://splunk.example.com:8088/services/collector/event
SEMAPHORE_AUDIT_SPLUNK_HEC_TOKEN='<HEC token>'
SEMAPHORE_AUDIT_SPLUNK_HEC_INDEX=security
SEMAPHORE_AUDIT_SPLUNK_HEC_CA_FILE=/etc/semaphore/siem-ca.pem

id, url and token are required, and the URL must start with https://. Use a different id than for Syslog. source and sourcetype default to semaphore and semaphore:audit. Certificates are checked the same way as for Syslog, and the standard HTTPS_PROXY and NO_PROXY variables apply.

Semaphore sends up to 100 events per request. The event field of each HEC event holds the audit event JSON, time is the event time, and host is the HA node ID, or the instance ID on a single node.

Restart Semaphore, then check that events arrive as described above.

How events are delivered​

  • If the receiver is down, events wait in the database and are sent when it is back. Users don't notice anything.
  • After network errors, restarts or an HA failover, some events can arrive twice. Use event_id to drop duplicates and seq to put events in order.
  • Over Syslog, if a connection breaks without an error, the event sent at that moment can be lost.
  • In an HA installation, one node at a time sends events to each destination. If Redis is unavailable, sending pauses, and events are still recorded.
  • Over HEC, an event counts as sent only after the receiver answers with a 2xx status. Any other answer, including 4xx, is retried. If the receiver crashes after answering, events it had not stored yet can be lost.

Over Syslog, each event is sent as an RFC 5424 Syslog message with the event JSON as its body. HOSTNAME is the HA node ID, or the instance ID on a single node, and MSGID is the event code.

The examples below are minimal and only show how to receive the events. They accept a connection from any client that can reach the port. In production, protect the receiver so that only your Semaphore servers can send events to it.

rsyslog example​

This rsyslog configuration accepts the TLS connection and writes one event per line:

global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/rsyslog.d/ca.pem"
  DefaultNetstreamDriverCertFile="/etc/rsyslog.d/cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/key.pem"
)

module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="anon")
input(type="imtcp" port="6514" ruleset="semaphore-audit")

template(name="semaphore-audit-json" type="string" string="%msg%\n")

ruleset(name="semaphore-audit") {
  action(type="omfile" file="/var/log/semaphore-audit.json" template="semaphore-audit-json")
}

Vector example​

This Vector configuration accepts the TLS connection, reads the event JSON and writes it to a file:

[sources.semaphore_audit]
type = "syslog"
mode = "tcp"
address = "0.0.0.0:6514"
tls.enabled = true
tls.crt_file = "/etc/vector/cert.pem"
tls.key_file = "/etc/vector/key.pem"

[transforms.semaphore_audit_event]
type = "remap"
inputs = ["semaphore_audit"]
source = ". = parse_json!(.message)"

[sinks.semaphore_audit_file]
type = "file"
inputs = ["semaphore_audit_event"]
path = "/var/log/semaphore-audit.json"
encoding.codec = "json"

Vector HEC example​

[sources.semaphore_audit_hec]
type = "splunk_hec"
address = "0.0.0.0:8088"
valid_tokens = ["<HEC token>"]
tls.enabled = true
tls.crt_file = "/etc/vector/cert.pem"
tls.key_file = "/etc/vector/key.pem"

[sinks.semaphore_audit_file]
type = "file"
inputs = ["semaphore_audit_hec"]
path = "/var/log/semaphore-audit.json"
encoding.codec = "json"

Splunk example​

Create a HEC token in Splunk (Settings → Data inputs → HTTP Event Collector), allow the security index for it, and set url to https://<splunk>:8088/services/collector/event. To find the events, search for index=security sourcetype="semaphore:audit".

Leave Enable indexer acknowledgement off for this token: Semaphore does not use it.

Monitor export​

When metrics are enabled, Semaphore Pro reports for each destination:

MetricMeaning
semaphore_audit_export_oldest_pending_secondsAge of the oldest event that is not sent yet, 0 when everything is sent
semaphore_audit_export_pending_eventsNumber of events waiting to be sent
semaphore_audit_export_errors_totalFailed send attempts

semaphore_audit_export_errors_total is counted by the node that sent, so add it up across nodes, for example sum by (destination) (increase(semaphore_audit_export_errors_total[15m])).

Each metric has a destination label with the destination id. To get an alert when the SIEM stops receiving events, watch the age of the oldest pending event:

- alert: SemaphoreAuditExportStalled
  expr: max by (destination) (semaphore_audit_export_oldest_pending_seconds) > 900
  for: 5m

Troubleshoot export​

  • Semaphore does not start. Check that both audit.syslog.id and audit.syslog.address are set, or audit.splunk_hec.id, url and token for HEC, and that the CA file contains PEM certificates.
  • The TLS connection fails. Check that the receiver certificate matches server_name and is signed by a CA that Semaphore trusts.
  • Events do not arrive. Check the Semaphore server log and the receiver log. After a failure, Semaphore waits a little before it tries again.
  • Some events arrive twice. This can happen after retries and failovers. Drop duplicates by event_id.
  • HEC answers 400, 401 or 403. Check the token, the indexes it is allowed to write to, and that indexer acknowledgement is off for it. The token never appears in the Semaphore log.

What is not recorded​

The semaphore command-line tool works with the database directly, so commands such as user add and user token are not recorded.

Some actions in the UI are not recorded: removing a license, app settings, clearing HA task state, Terraform inventory aliases, deleting a Terraform state, workflow runs and project invitations. Template descriptions, views, clearing the project cache and scheduled secret storage syncs are not recorded either.

What's next​