Bazel Operations and Debugging

Remote Execution and Caching

The idea

Bazel’s Execution phase runs actions (build commands) one by one. Each action is a pure function: given these input files and this command line, produce these output files. If two actions have identical inputs and commands, they produce identical outputs. This means:

1. Caching: If you’ve run an action before (same inputs, same command), skip it and reuse the previous output.
2. Remote caching: Share this cache across machines. CI builds an action; every developer gets the cached result.
3. Remote execution: Ship the action’s inputs to a remote worker, let it run the command, get the outputs back. The worker doesn’t need anything installed — Bazel sends the toolchain binaries too.

All three depend on the same property: actions are hermetic — they only read declared inputs and only produce declared outputs. The sandbox (see Execution phase) enforces this locally. Remote execution enforces it structurally — the worker literally only has the declared inputs.

Remote caching (the easy win)

You can add shared caching with one line in .bazelrc:

build:cache --remote_cache=grpc://cache.example.com:443

Actions still run locally, but their results are stored in and fetched from a shared cache. CI populates the cache; developers get cache hits for unchanged dependencies. This is the highest-ROI optimization for most teams.

Full remote execution

build:remote --remote_executor=grpc://remote.example.com:443
build:remote --remote_cache=grpc://remote.example.com:443
build:remote --jobs=200  # many actions in parallel on the cluster

Actions are shipped to remote workers. The protocol uses two storage layers:

• Content Addressable Storage (CAS): files stored by content hash. Identical files are deduplicated.
• Action Cache (AC): maps (command + input hashes) → output hashes. A hit means no execution needed.

---

Debugging Bazel Builds

Bazel has three query tools, each operating on a different phase’s output. Knowing which to use depends on the kind of bug you’re chasing.

bazel query — The target graph (Loading output)

Knows what targets exist and their declared dependencies. Does NOT know about configurations, select() resolution, or actions. Use for structural questions.

bazel query "deps(//rust/my_module:my_module)"              # what does it depend on?
bazel query "rdeps(//..., @crate_index//:pyo3)"              # what depends on pyo3?
bazel query "//rust/my_module:my_module_rs" --output=label_kind  # what rule type?
bazel query "//rust/my_module:*"                             # all targets in package

bazel cquery — The configured target graph (Analysis output)

Knows about configurations and select() resolution. Use when you need to see how platform-dependent logic resolved.

bazel cquery "deps(//rust/my_module:my_module)" --output=label_kind
bazel cquery "//target" --output=starlark --starlark:expr="providers(target)['CrateInfo'].type"

bazel aquery — The action graph (Analysis output)

Shows the actual commands Bazel will run: executable, arguments, inputs, outputs. This is the most powerful debugging tool. If a compiler invocation has the wrong flags, you’ll see it here.

bazel aquery "//rust/my_module:my_module_rs"
# Shows: mnemonic, inputs, outputs, full command line, environment variables

Seeing commands as they execute

bazel build //target --subcommands     # prints every command before running it
bazel build //target --sandbox_debug   # preserves sandbox dirs for inspection

If an action fails with “file not found,” --sandbox_debug lets you inspect the sandbox to see what files were (and weren’t) present. The missing file is an undeclared input.

Matching errors to phases

The first step in debugging is figuring out which phase the error comes from:

Error looks like	Phase	What to do
syntax error in BUILD file	Loading	Read the error. Check .bzl files.
no such target '@crate_index//:pyo3'	Fetch/Loading	bazel fetch, check MODULE.bazel
doesn't have mandatory providers: 'CrateInfo'	Analysis	Wrong target type in deps. Use bazel query --output=label_kind
no matching toolchain found	Analysis	Check register_toolchains() and platform constraints
failed to resolve: use of undeclared crate	Execution	Rust compiler error. Use bazel aquery to inspect --extern flags
ld: symbol(s) not found	Execution	Linker error. Check extension-module feature, linker flags
ModuleNotFoundError	Runtime	.so missing or wrong filename. Check runfiles, imports attribute

---

Build Performance

Profiling

bazel build //target --profile=/tmp/profile.json.gz   # collect profile
bazel analyze-profile /tmp/profile.json.gz             # summarize
# or open in Chrome: chrome://tracing/

Shows time per phase, time per action, the critical path (longest chain of sequential actions — the bottleneck even with infinite parallelism), and parallelism over time.

bazel build //target --execution_log_json_file=/tmp/exec_log.json

Execution log shows per-action cache hit rates (remoteCacheHit), wall time, and input file hashes. Use this to find actions with poor cache hit rates.

Common problems and solutions

Problem	Cause	Fix
Everything rebuilds on one file change	Widely-depended-on target has unnecessary dep	bazel query "somepath(//changed, //rebuilt)"
Build is slow despite parallelism	Long critical path (deep sequential chain)	Break up large libraries
Poor remote cache hit rate	Non-hermetic actions, volatile inputs	Check execution log for culprits

PyO3-specific tips

• Split large Rust crates. One crate = one compilation action with no internal parallelism.
• Use abi3. Don’t rebuild the extension for each Python version.
• Verify incrementality. If only Python code changes, the .so shouldn’t rebuild. Check with bazel aquery.

---

Visibility

Every target has a visibility attribute controlling who can depend on it. The default is private (same package only).

"//visibility:public"              # anyone
"//visibility:private"             # same package only (default)
"//python/my_app:__pkg__"          # only targets in //python/my_app/
"//python:__subpackages__"         # //python/ and all sub-packages

Recommended pattern for PyO3: Keep Rust internals private. Expose only the py_library wrapper:

package(default_visibility = ["//visibility:private"])
 
rust_shared_library(name = "my_module_rs", ...)  # private
genrule(name = "my_module_rename", ...)          # private
 
py_library(
    name = "my_module",
    visibility = ["//python:__subpackages__"],    # public API
)

This ensures consumers use the py_library (which handles runfiles and sys.path correctly), not the raw .so.