Documentation Grafana Alloy Reference Components loki loki.source.file

loki.source.file

loki.source.file reads log entries from files and forwards them to other loki.* components. New log entries are forwarded whenever a log entry line ends with the \n character.

You can specify multiple loki.source.file components by giving them different labels.

Usage

Alloy Copy

loki.source.file "<LABEL>" {
  targets    = <TARGET_LIST>
  forward_to = <RECEIVER_LIST>
}

Arguments

The component starts a new reader for each of the given targets and fans out log entries to the list of receivers passed in forward_to.

You can use the following arguments with loki.source.file:

The encoding argument must be a valid IANA encoding name. If not set, it defaults to UTF-8.

You can use the tail_from_end argument when you want to tail a large file without reading its entire content. When set to true, only new logs are read, ignoring the existing ones.

The legacy_positions_file argument is used when you are transitioning from Grafana Agent Static Mode to Grafana Alloy. The format of the positions file is different in Grafana Alloy, so this will convert it to the new format. This operation only occurs if the new positions file doesn’t exist and the legacy_positions_file is valid. When legacy_positions_file is set, Alloy will try to find previous positions for a given file by matching the path and labels, falling back to matching on path only if no match is found.

If you want to read a UTF-16 file with a Byte Order Mark (BOM), set encoding to UTF-16. BOMs will be ignored if encoding is set to either UTF-16BE or UTF-16LE.

Blocks

You can use the following blocks with loki.source.file:

decompression

The decompression block contains configuration for reading logs from compressed files. The following arguments are supported:

If you compress a file under a folder being scraped, loki.source.file might try to ingest your file before you finish compressing it. To avoid it, pick an initial_delay that’s long enough to avoid it.

Currently supported compression formats are:

• gz - for Gzip
• z - for zlib
• bz2 - for bzip2

The component can only support one compression format at a time. To handle multiple formats, you must create multiple components.

file_watch

The file_watch block configures how often log files are polled from disk for changes. The following arguments are supported:

If no file changes are detected, the poll frequency doubles until a file change is detected or the poll frequency reaches the max_poll_frequency.

If file changes are detected, the poll frequency is reset to min_poll_frequency.

file_match

The file_match block enables built-in file discovery directly within loki.source.file. When enabled, the component discovers files on the local filesystem using glob patterns and the doublestar library.

The following arguments are supported:

Benefits of using file_match over local.file_match:

• Simplified configuration: No need for a separate local.file_match component
• Reduced overhead: File discovery is integrated directly into loki.source.file, eliminating the need for component-to-component communication
• Same functionality: Supports the same glob patterns and features, including __path_exclude__ for excluding files

When enabled is set to true, you can use glob patterns, for example, /tmp/*.log or /var/log/**/*.log, directly in the targets argument’s __path__ label. The component periodically scans the filesystem based on sync_period and automatically discovers new files, removes deleted files, and ignores files older than ignore_older_than if specified.

Exported fields

loki.source.file doesn’t export any fields.

Component health

loki.source.file is only reported as unhealthy if given an invalid configuration.

Debug information

loki.source.file exposes some target-level debug information per reader:

• The tailed path.
• Whether the reader is running.
• The last recorded read offset in the positions file.

Debug metrics

• loki_source_file_encoding_failures_total (counter): Number of encoding failures.
• loki_source_file_file_bytes_total (gauge): Number of bytes total.
• loki_source_file_files_active_total (gauge): Number of active files.
• loki_source_file_read_bytes_total (gauge): Number of bytes read.
• loki_source_file_read_lines_total (counter): Number of lines read.

Component behavior

If the decompression feature is deactivated, the component continuously monitors and tails the files. The component remains active after reaching the end of a file, and reads new entries in real-time as they’re appended to the file.

Each element in the list of targets as a set of key-value pairs called labels. The set of targets can either be static, or dynamically provided periodically by a service discovery component. The special label __path__ must always be present and must contain the absolute path of the file to read from.

The __path__ value is available as the filename label to each log entry the component reads. All other labels starting with a double underscore are considered internal and are removed from the log entries before they’re passed to other loki.* components.

The component uses its data path, a directory named after the domain’s fully qualified name, to store its positions file. The positions file stores read offsets, so that if a component or Alloy restarts, loki.source.file can pick up tailing from the same spot.

The data path is inside the directory configured by the --storage.path command line argument.

If a file is removed from the targets list, its positions file entry is also removed. When it’s added back on, loki.source.file starts reading it from the beginning.

Examples

The following examples demonstrate how you can collect log entries with loki.source.file.

Static targets

This example collects log entries from the files specified in the targets argument and forwards them to a loki.write component.

Alloy Copy

loki.source.file "tmpfiles" {
  targets    = [
    {__path__ = "/tmp/foo.txt", "color" = "pink"},
    {__path__ = "/tmp/bar.txt", "color" = "blue"},
    {__path__ = "/tmp/baz.txt", "color" = "grey"},
  ]
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

File globbing with file_match (recommended)

This example collects log entries from files matching the *.log pattern using the built-in file_match block. When files appear or disappear, the list of targets is automatically updated based on the sync_period configuration.

Alloy Copy

loki.source.file "tmpfiles" {
  targets    = [
    {__path__ = "/tmp/*.log"},
  ]
  forward_to = [loki.write.local.receiver]
  file_match {
    enabled = true
    sync_period = "10s"
  }
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

File globbing with local.file_match

This example demonstrates the alternative approach using a separate local.file_match component. This approach is still supported but using the file_match block is recommended for better performance and simpler configuration.

Alloy Copy

local.file_match "logs" {
  path_targets = [
    {__path__ = "/tmp/*.log"},
  ]
}

loki.source.file "tmpfiles" {
  targets    = local.file_match.logs.targets
  forward_to = [loki.write.local.receiver]
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

Decompression

This example collects log entries from compressed files matching the *.gz pattern using the built-in file_match block with decompression enabled.

Alloy Copy

loki.source.file "tmpfiles" {
  targets    = [
    {__path__ = "/tmp/*.gz"},
  ]
  forward_to = [loki.write.local.receiver]
  file_match {
    enabled = true
    sync_period = "10s"
  }
  decompression {
    enabled       = true
    initial_delay = "10s"
    format        = "gz"
  }
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

Compatible components

loki.source.file can accept arguments from the following components:

• Components that export Targets
• Components that export Loki LogsReceiver