Log Query Language

This document describes at a high level, the log query language used to query and filter API request and response data.

Overview

Log Query Box
Log Query Box

The Log Query Language in the APIToolkit Dashboard is a robust tool designed for querying and filtering API request and response data. It allows users to perform detailed queries, group data, and create metrics based on specific criteria in the request/response logs.

A query is a boolean expression that specifies a subset of all the request/response log entries. Queries can range from simple restrictions to complex conjunctive/disjunctive expressions. Examples include:

  • Simple restriction: method="GET"
  • Conjunctive restriction: method="GET" AND status_code=404
  • Disjunctive restriction: method="GET" OR method="PUT"
  • Complex conjunctive/disjunctive expression: method="GET" AND (response_body.value="value" OR status_code=302)

Syntax notation

The following sections provide an overview of the log query language syntax, and discuss in detail how queries are structured and how matching is performed.

The Log query language syntax can be seen in terms of queries and comparisons.

A query is a string/text containing an expression:

expression = comparison { ("AND" | "OR") ["NOT"] comparison }

A comparison is usually a boolean operation such as

status_code=200

It is usually of the shape [FIELD_PATH] [OPERATOR] [VALUE]

  • [FIELD_PATH] is a field in a request/response log entry eg status_code, or response_body.message (Assuming there’s a message field in the response_body)
  • [OPERATOR] is a comparison operator and should be one of the following:
    • = : Equal
    • != : Not Equal
    • > : Greater than
    • < : Less than
    • >= : Greater than or Equal
    • <= : Less than or equal
    • =~ : Regular expression search for a pattern
    • !~ : Regular expression search not for a pattern
    • : : substring matching on the log entry field. ie does this field contain …
  • [VALUE] is a number or string or a null or boolean which we are comparing the content of the [FIELD_PATH] against.
    • Numbers (integers and floats) eg 1, 2, 3, 1.0, 22.2, 10000, etc
    • Strings are always quoted via double quotes eg “Apitoolkit”, “123”, etc
    • Boolean are case TRUE or FALSE or true or false
    • Null is an unquoted null text. ie null

Field Path Identifiers

The field identifier defines the path from the log entry root to the relevant field. It’s syntax is mildly inspired by the jq syntax

Field path identifiers define the path from the log entry root to the relevant field, inspired by the jq syntax. For instance, status_code or query_param.key for nested fields. For nested fields within arrays, use [], e.g., response_body.addresses.[].street_number.

Log Entry
Log Entry

The field paths must always reference the fields in the log entry. Eg in the example above, I could reference the status_code via status_code. I could reference nested fields in objects by using a . as the separator. Eg for the key field in query_param, I could do query_param.key eg

query_param.key = ""

For nested fields with arrays, we can use [] as a special character to signify that that section of the field path is an array or list. Eg

response_body.addresses.[].street_number = 20

This example above assumes that there is an addresses array and in that array each item is a object with a field street_number, and we want the results where that street_number is 20

Default field path identifiers

These are fields which are always available and are usually at the root of the log entry, which you can query by.

  • raw_url: The raw URL of the given request, excluding the host. It includes the full URL path and the query parameters without the host. eg /test/path/1?q=value
  • url_path: This is the url_path section but the path parameters are not substituted. Eg /test/path/{id}. This doesn’t include the query parameters
  • method: The HTTP method for the given request
  • created_at: When the request happened
  • id: A unique ID generated by apitoolkit for every request log entry
  • host: The host which the request came from. Eg https://apitoolkit.io
  • referer: The referer for the request. This is gotten from the referer http header
  • status_code: The HTTP status code of the given request as a number
  • request_headers: A key value pair, usually of a key to a list of values. This represents all the http request headers which came with that request.
  • response_headers: A key value pair, usually of a key to a list of values. This represents all the http response headers which came with that response.
  • request_body: Usually an object representing the parsed request json (or text) for that given request. You can use this as part of your log queries.
  • response_body: Usually an object representing the parsed response json (or text) for that given response. You can use this as part of your log queries.

Using the Log Query Language

When using the Log Query Language in the Logs Explorer, you can effectively query and analyze your API traffic, gaining insights into usage patterns, troubleshooting issues, and understanding user interactions with your API.