The Intelligence Report Nobody Can Explain

It's 0347 hours. A CAF intelligence analyst receives a real-time fused assessment from a multi-modal AI system: a ground vehicle on Route Corridor Bravo has been classified as "high-confidence hostile combatant vehicle — 91%."

The Rules of Engagement for the forward patrol require positive identification before engagement. The analyst stares at the assessment. It's a clean output — classification, confidence score, bounding box overlay, track history.

But nothing explains why.

What sensor contributed most to that 91%? Was it the RF signature, the thermal profile, or the silhouette match? Did the model spot features consistent with a known adversary platform, or is this an anomaly-driven classification based on outlier behaviour? Did any modality disagree? What changed in the last 60 seconds that pushed the confidence from 73% to 91%?

The system can't answer those questions. The model is a black box.

The patrol holds fire. The target moves deeper. Thirty minutes later, a friendly logistics vehicle takes an IED strike on Route Corridor Bravo.

This isn't a story about AI failure. It's a story about AI accountability failure. The system might have been right. We'll never know — because nobody can reconstruct how it reached its conclusion.

The Compliance Floor

The Department of National Defence's AI Ethics Policy is unambiguous: algorithmic systems used in defence must be transparent, explainable, and auditable. It's not aspirational language — it's policy. Similar requirements exist across NATO allies under frameworks like the European Commission's AI Act (defence carve-outs notwithstanding) and the U.S. DoD's Responsible AI Strategy.

For procurement officers, this creates a concrete evaluation criterion. If a vendor's AI system can't explain its outputs, it doesn't meet the baseline for ethical deployment. Period.

But compliance is the floor, not the ceiling. Many programs treat explainability as a documentation problem — a PDF of model architecture and training data provenance that sits in a compliance binder. That's necessary but insufficient. Real explainability means that the operator, the analyst, and the post-mission auditor can interrogate the model's reasoning at the point of use.

The audit trail problem: When an AI-influenced intelligence product contributes to an operational decision, the after-action review needs to reconstruct that decision chain. "The model said 91%" is not an audit trail. It's a dead end. DND policy requires traceable reasoning — and traceable reasoning requires explainable AI.

Why Explainability Is an Operational Requirement

Beyond compliance, explainable AI solves three hard operational problems:

1. Debugging model errors in the field

When a deployed model produces an unexpected output, the operator needs to know why — not to override the AI, but to understand whether the error is a model problem, a sensor problem, or a genuine novel situation. Without explainability, every unexpected output requires manual analyst review. With explainability, the system can show which features drove the decision, letting the operator quickly determine: the model focused on the wrong artifact (fix the model), a sensor is degrading (fix the sensor), or this is a genuinely new class of object (adapt the concept).

2. Building operator trust

Trust isn't a training exercise. It's built through repeated, transparent interaction with the system. When operators can see why the AI reached a conclusion, they calibrate their own reliance appropriately — trusting strong signals, questioning edge cases, and developing the instinct for when the system's reasoning is sound versus when it's grasping at noise.

Black-box systems erode trust. Every unexplained 91% makes the next 93% harder to believe.

3. After-action reviews and continuous improvement

Post-mission analysis requires understanding which AI decisions were correct, which were incorrect, and why. This feedback loop is what drives model improvement between deployments. Without explainability, you can evaluate whether the model was right or wrong — but you can't learn why, which means you can't reliably improve it.

Technical Approaches to Explainable AI

Explainable AI isn't a single technique. It's a layered approach:

Attention Visualization

Attention mechanisms in neural networks produce attention weight maps — showing which parts of the input the model focused on when making its decision. In multi-modal fusion, cross-attention between modalities reveals which sensor channels contributed most to a fused classification. An operator seeing a detection can immediately see: "The RF channel contributed 60% of the decision weight, EO contributed 25%, and text intelligence contributed 15%." This is intuitive, fast, and actionable.

SHAP Values (SHapley Additive exPlanations)

SHAP provides a mathematically principled way to attribute the model's output to individual input features. For processed intelligence products, SHAP scores reveal which specific features — frequency band, vehicle thermal signature, silhouette dimensions — drove how much margin toward the model's confidence score. Unlike simpler feature importance methods, SHAP values satisfy fairness properties: they guarantee that feature attributions add up to the model's actual prediction.

Counterfactual Explanations

A counterfactual explanation answers: "What would have needed to change for the model to reach a different conclusion?" For example: "If the thermal signature had been 4°C cooler, the classification would shift from hostile combatant to civilian logistics vehicle." This is particularly powerful for edge cases and ambiguous classifications — giving operators a concrete threshold for understanding model sensitivity.

Provenance and Lineage Tracking

At the system level, every AI-generated output should carry a provenance graph — a record of the input data, the model version, the intermediate representations, and the reasoning chain that produced the final output. In defence AI, this becomes the audit trail that satisfies after-action review requirements and maintains institutional trust across personnel rotations.

NovaFuse's Approach: Interpretable by Design

At NovaFuse, we treat explainability not as a post-hoc add-on but as an architectural requirement. From the earliest stages of our IDEaS research, we've designed our multi-modal fusion pipeline to produce traceable, auditable outputs at every stage.

Our attention-based fusion architecture is inherently more interpretable than black-box ensemble approaches — the attention weights themselves become a first-class explainability signal. We layer SHAP-based feature attribution on top of our classification outputs, giving operators feature-level visibility into every decision.

And our provenance framework captures the full lineage of every fused intelligence product: sensor inputs, intermediate model representations, confidence scores, and reasoning summaries. It's not compliance documentation — it's an operational tool.

We believe the next generation of defence AI won't just be accurate. It will have to be defensible in the literal sense — able to defend its reasoning to the operator, the commander, and the auditor.

Your Program Needs a Strategy for Explainable AI

Whether you're scoping a new intelligence system, evaluating AI vendors, or drafting requirements for your next RFP, explainable AI needs to be on the table — not as a checkbox but as a functional requirement.

We're happy to walk your team through what XAI looks like in practice for defence applications: from attention visualization on live sensor feeds to SHAP-driven model auditing to provenance-based audit trails.

Contact NovaFuse for a technical briefing on explainable AI for your program.

info@novafuse.ca