Capsula
Capsula, a Latin word meaning box, is a Python package designed to help researchers and developers easily capture their command/function execution context for reproducibility.
With Capsula, you can capture:
- CPU information with
CpuContext
- Python version with
PlatformContext
- Current working directory with
CwdContext
- Git repository information (commit hash, branch, etc.) with
GitRepositoryContext
- Output of shell commands (e.g.,
poetry check --lock
) withCommandContext
- Files (e.g., output files,
pyproject.toml
,requirements.txt
) withFileContext
- Arguments of Python functions with
FunctionContext
- Environment variables with
EnvVarContext
- Uncaught exceptions with
UncaughtExceptionWatcher
- Execution time with
TimeWatcher
The captured contexts are dumped into JSON files for future reference and reproduction.
Usage example
For project-wide settings, prepare a capsula.toml
file in the root directory of your project. An example of the capsula.toml
file is as follows:
[pre-run]
contexts = [
{ type = "CwdContext" },
{ type = "CpuContext" },
{ type = "PlatformContext" },
{ type = "GitRepositoryContext", name = "capsula", path = ".", path_relative_to_project_root = true },
{ type = "CommandContext", command = "poetry check --lock", cwd = ".", cwd_relative_to_project_root = true },
{ type = "FileContext", path = "pyproject.toml", copy = true, path_relative_to_project_root = true },
{ type = "FileContext", path = "poetry.lock", copy = true, path_relative_to_project_root = true },
{ type = "CommandContext", command = "pip freeze --exclude-editable > requirements.txt", cwd = ".", cwd_relative_to_project_root = true },
{ type = "FileContext", path = "requirements.txt", move = true, path_relative_to_project_root = true },
{ type = "EnvVarContext", name = "HOME" },
]
reporters = [{ type = "JsonDumpReporter" }]
[in-run]
watchers = [{ type = "UncaughtExceptionWatcher" }, { type = "TimeWatcher" }]
reporters = [{ type = "JsonDumpReporter" }]
[post-run]
reporters = [{ type = "JsonDumpReporter" }]
Then, all you need to do is decorate your Python function with the @capsula.run()
decorator. You can also use the @capsula.context()
decorator to add a context specific to the function.
The following is an example of a Python script that estimates the value of π using the Monte Carlo method:
import random
import capsula
@capsula.run()
@capsula.context(capsula.FunctionContext.builder(), mode="pre")
@capsula.context(capsula.FileContext.builder("pi.txt", move=True), mode="post")
def calculate_pi(n_samples: int = 1_000, seed: int = 42) -> None:
random.seed(seed)
xs = (random.random() for _ in range(n_samples))
ys = (random.random() for _ in range(n_samples))
inside = sum(x * x + y * y <= 1.0 for x, y in zip(xs, ys))
# You can record values to the capsule using the `record` method.
capsula.record("inside", inside)
pi_estimate = (4.0 * inside) / n_samples
print(f"Pi estimate: {pi_estimate}")
capsula.record("pi_estimate", pi_estimate)
print(f"Run name: {capsula.current_run_name()}")
with open("pi.txt", "w") as output_file:
output_file.write(f"Pi estimate: {pi_estimate}.")
if __name__ == "__main__":
calculate_pi(n_samples=1_000)
After running the script, a directory (calculate_pi_20240711_190108_6I2M
in this example) will be created under the <project-root>/vault
directory, and you will find the output files in the directory:
$ tree vault/calculate_pi_20240711_190108_6I2M
vault/calculate_pi_20240711_190108_6I2M
├── in-run-report.json # Generated by the `JsonDumpReporter` in `capsula.toml` (`in-run` section)
├── pi.txt # Moved by the `FileContext` specified with the decorator in the script
├── poetry.lock # Copied by the `FileContext` specified in `capsula.toml` (`pre-run` section)
├── post-run-report.json # Generated by the `JsonDumpReporter` in `capsula.toml` (`post-run` section)
├── pre-run-report.json # Generated by the `JsonDumpReporter` in `capsula.toml` (`pre-run` section)
├── pyproject.toml # Copied by the `FileContext` specified in `capsula.toml` (`pre-run` section)
└── requirements.txt # Moved by the `FileContext` specified in `capsula.toml` (`pre-run` section)
The contents of the JSON files are as follows:
Example of output pre-run-report.json
:
{
"cwd": "/home/nomura/ghq/github.com/shunichironomura/capsula",
"cpu": {
"python_version": "3.8.19.final.0 (64 bit)",
"cpuinfo_version": [
9,
0,
0
],
"cpuinfo_version_string": "9.0.0",
"arch": "X86_64",
"bits": 64,
"count": 12,
"arch_string_raw": "x86_64",
"vendor_id_raw": "GenuineIntel",
"brand_raw": "Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz",
"hz_advertised_friendly": "2.9000 GHz",
"hz_actual_friendly": "2.9040 GHz",
"hz_advertised": [
2900000000,
0
],
"hz_actual": [
2904008000,
0
],
"stepping": 5,
"model": 165,
"family": 6,
"flags": [
"3dnowprefetch",
"abm",
"adx",
"aes",
"apic",
"arch_capabilities",
"arch_perfmon",
"avx",
"avx2",
"bmi1",
"bmi2",
"clflush",
"clflushopt",
"cmov",
"constant_tsc",
"cpuid",
"cx16",
"cx8",
"de",
"ept",
"ept_ad",
"erms",
"f16c",
"flush_l1d",
"fma",
"fpu",
"fsgsbase",
"fxsr",
"ht",
"hypervisor",
"ibpb",
"ibrs",
"ibrs_enhanced",
"invpcid",
"invpcid_single",
"lahf_lm",
"lm",
"mca",
"mce",
"md_clear",
"mmx",
"movbe",
"msr",
"mtrr",
"nopl",
"nx",
"osxsave",
"pae",
"pat",
"pcid",
"pclmulqdq",
"pdcm",
"pdpe1gb",
"pge",
"pni",
"popcnt",
"pse",
"pse36",
"rdrand",
"rdrnd",
"rdseed",
"rdtscp",
"rep_good",
"sep",
"smap",
"smep",
"ss",
"ssbd",
"sse",
"sse2",
"sse4_1",
"sse4_2",
"ssse3",
"stibp",
"syscall",
"tpr_shadow",
"tsc",
"vme",
"vmx",
"vnmi",
"vpid",
"x2apic",
"xgetbv1",
"xsave",
"xsavec",
"xsaveopt",
"xsaves",
"xtopology"
],
"l3_cache_size": 12582912,
"l2_cache_size": "1.5 MiB",
"l1_data_cache_size": 196608,
"l1_instruction_cache_size": 196608,
"l2_cache_line_size": 256,
"l2_cache_associativity": 6
},
"platform": {
"machine": "x86_64",
"node": "SHUN-DESKTOP",
"platform": "Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.34",
"release": "5.15.153.1-microsoft-standard-WSL2",
"version": "#1 SMP Fri Mar 29 23:14:13 UTC 2024",
"system": "Linux",
"processor": "x86_64",
"python": {
"executable_architecture": {
"bits": "64bit",
"linkage": "ELF"
},
"build_no": "default",
"build_date": "Jul 7 2024 07:23:53",
"compiler": "GCC 11.4.0",
"branch": "",
"implementation": "CPython",
"version": "3.8.19"
}
},
"git": {
"capsula": {
"working_dir": "/home/nomura/ghq/github.com/shunichironomura/capsula",
"sha": "a308c82bf9c8670de62d155e83ebf78f816f7851",
"remotes": {
"origin": "ssh://git@github.com/shunichironomura/capsula.git"
},
"branch": "main",
"is_dirty": false,
"diff_file": null
}
},
"command": {
"poetry check --lock": {
"command": "poetry check --lock",
"cwd": "/home/nomura/ghq/github.com/shunichironomura/capsula",
"returncode": 0,
"stdout": "All set!\n",
"stderr": ""
},
"pip freeze --exclude-editable > requirements.txt": {
"command": "pip freeze --exclude-editable > requirements.txt",
"cwd": "/home/nomura/ghq/github.com/shunichironomura/capsula",
"returncode": 0,
"stdout": "",
"stderr": ""
}
},
"file": {
"/home/nomura/ghq/github.com/shunichironomura/capsula/pyproject.toml": {
"copied_to": [
"/home/nomura/ghq/github.com/shunichironomura/capsula/vault/calculate_pi_20240711_190108_6I2M/pyproject.toml"
],
"moved_to": null,
"hash": {
"algorithm": "sha256",
"digest": "bfd58ba4947798d61ae3679cf3b06def700aefcdc33a2e4935164f480f16191c"
}
},
"/home/nomura/ghq/github.com/shunichironomura/capsula/poetry.lock": {
"copied_to": [
"/home/nomura/ghq/github.com/shunichironomura/capsula/vault/calculate_pi_20240711_190108_6I2M/poetry.lock"
],
"moved_to": null,
"hash": {
"algorithm": "sha256",
"digest": "210e74a7cd2db48b95dee0def67d5bbba33a86ab85859bea6c17ca74b48a2448"
}
},
"/home/nomura/ghq/github.com/shunichironomura/capsula/requirements.txt": {
"copied_to": [],
"moved_to": "/home/nomura/ghq/github.com/shunichironomura/capsula/vault/calculate_pi_20240711_190108_6I2M",
"hash": {
"algorithm": "sha256",
"digest": "7d8d12ce44cae648c0f7cc7a636e226c857510276cd3a221de1ffa4d7125c5b0"
}
}
},
"env": {
"HOME": "/home/nomura"
},
"function": {
"calculate_pi": {
"file_path": "examples/simple_decorator.py",
"first_line_no": 6,
"bound_args": {
"n_samples": 1000,
"seed": 42
}
}
}
}
Example of output in-run-report.json
:
{
"inside": 782,
"pi_estimate": 3.128,
"time": {
"execution_time": "0:00:00.000612"
},
"exception": {
"exception": {
"exc_type": null,
"exc_value": null,
"traceback": null
}
}
}
Example of output post-run-report.json
:
{
"file": {
"pi.txt": {
"copied_to": [],
"moved_to": "/home/nomura/ghq/github.com/shunichironomura/capsula/vault/calculate_pi_20240711_190108_6I2M",
"hash": {
"algorithm": "sha256",
"digest": "a64c761cb6b6f9ef1bc1f6afa6ba44d796c5c51d14df0bdc9d3ab9ced7982a74"
}
}
}
}
Installation
You can install Capsula via pip:
Or via conda:
Licensing
This project is licensed under the terms of the MIT.
Additionally, this project includes code derived from the Python programming language, which is licensed under the Python Software Foundation License Version 2 (PSF-2.0). For details, see the LICENSE file.