Task Logging¶
Rocketry’s logging system is based on logging library (standard library) and on Red Bird. Rocketry uses Logging’s logging mechanisms and extend them with Red Bird in order to read from the logs.
Mechanism¶
There is a logger called rocketry.task
which is
used to log the tasks’ actions. This logger should
have one repo handler (redbird.logging.RepoHandler
)
which logs the task actions to a repository that
can be read using Red Bird’s unified query syntax.
The logs, appart from failure, are logged with
level INFO
thus the logger should not filter
this level off.
Types of task actions:
run
: Logged when a task starts.success
: Logged when a task finishes without exceptions.fail
: Logged when a task finishes with an error.terminate
: Logged when a task is terminated before it finished.inaction
: Special action to indicate the task did nothing. Requires raising a special exception.crash
: The task had previously silently crashed. This is to indicate that the task is no longer running.
Log Record¶
Each log record in the repository should contain at least the following fields or attributes:
created
: Timestamp when the log record was created (Logging library creates).task_name
: Name of the task.action
: What the log was about. Either run, success, fail, terminate or inaction.
It is also recommended to have fields/attributes:
run_id
: Identifier of the run. Relevant if multilaunch used.
You may add any other field or attribute from what the Logging library creates or create attributes your own.
Here is a minimal example of a log record model that Red Bird accepts:
from pydantic import BaseModel
class MinimalRecord(BaseModel):
task_name: str
action: str
created: float
Log record models are passed to the repo handlers to specify the data format in which our logs are. We take a look into this in a bit.
Here are some premade log record models you may use:
rocketry.log.MinimalRecord
: Bare minimum for the logging to work.rocketry.log.LogRecord
: Has the typical elements of logging.LogRecord and extras required by rocketry.rocketry.log.TaskLogRecord
: Has the same asLogRecord
but also includes start, end and runtimes.
There are also variants that include field run_id
:
rocketry.log.MinimalRunRecord
rocketry.log.RunRecord
rocketry.log.TaskRunRecord
Setting Up Repo to a Logger¶
By default, Rocketry creates a repo handler with MemoryRepo
in it. This handler logs the records only to an in-memory Python
list that is not maintained when the interpreter is closed.
You may want to log the records to disk in order to maintain persistence in scheduler’s state in case of restart or shutdown.
First, we fetch the task logger:
import logging
logger = logging.getLogger('rocketry.task')
Here is an example to log to a CSV file:
from rocketry.log import MinimalRecord
from redbird.repos import CSVFileRepo
from redbird.logging import RepoHandler
# Creating the repo
repo = CSVFileRepo(filename="path/to/repo.csv", model=MinimalRecord)
# Adding the repo to the logger
logger = logging.getLogger('rocketry.task')
handler = RepoHandler(repo=repo)
logger.addHandler(handler)
Another common pattern is to log the records to a SQL database. This can be done with SQLAlchemy:
from redbird.repos import SQLRepo
from sqlalchemy import create_engine
engine = create_engine("sqlite:///app.db")
repo = SQLRepo(engine=engine, table="tasks", if_missing="create", model=MinimalRecord, id_field="created")
handler = RepoHandler(repo=repo)
logger.addHandler(handler)
Read more about repositories from Red Bird’s documentation.
Querying the Logger¶
Here is an illustration of getting the repository:
import logging
logger = logging.getLogger('rocketry.task')
for handler in logger.handlers:
if hasattr(handler, "repo"):
break
repo = handler.repo
Then we can query this repo:
repo.filter_by(task_name="my_task", action="run").all()
The task_name
is already injected if you
call the logger in a task. Tasks use a TaskAdapter
that does this trick:
@app.task()
def do_things():
...
task_logger = app.session['do_things'].logger
task_logger.filter_by(action="run").all()
Read more about querying from Red Bird’s documentation.