Understanding Bayesian Network
A Bayesian network answers the question of how an agent can hold beliefs about many variables at once without writing down a joint distribution whose size grows exponentially with their number. It does so by recording only direct influences: an edge from X to Y says that X is a parent of Y, and each node stores P(node | parents) as a table with one row per combination of parent values. Russell and Norvig introduce the idea with a burglar alarm that responds to burglaries and, less reliably, to earthquakes, and two neighbours who call when they hear it. Five variables, four edges, and ten numbers replace a joint table of thirty-one.
The semantics is a single equation: the probability of a complete assignment to all the variables is the product of the conditional probabilities read from the tables. For the alarm sounding with no burglary and no earthquake while both neighbours call, that product is 0.90 × 0.70 × 0.001 × 0.999 × 0.998 = 0.000628. Because every joint entry can be recovered this way, the network is not an approximation of the joint distribution; it is the joint distribution, stored compactly.
The compactness rests on conditional independence. Comparing the product with the chain rule of probability shows that the network is a correct model exactly when each variable is conditionally independent of its other predecessors given its parents, for some ordering that places parents before children. Two consequences follow from the graph alone: a node is independent of its non-descendants given its parents, and a node is independent of every other node given its Markov blanket, the set of its parents, children, and children’s other parents. Building the network in causal order keeps it small; building it in the wrong order forces extra edges and harder-to-assess numbers.
A query asks for the posterior of one variable given evidence on others, and every such query is a normalised sum of products of table entries over the hidden variables. Enumeration evaluates that sum directly and repeats work; variable elimination stores intermediate results as factors and combines them by pointwise product and summing out. On a polytree, a graph with at most one undirected path between any two nodes, elimination runs in time linear in the number of table entries. On multiply connected graphs the factors can grow exponentially, and the general problem is as hard as counting the satisfying assignments of a propositional formula. Approximate methods - rejection sampling, likelihood weighting, and Markov chain Monte Carlo - trade exactness for consistent estimates whose cost depends on how the evidence is handled.
How to Calculate
P(x₁, …, xₙ) = Πᵢ P(xᵢ | parents(Xᵢ))
where
- x₁, …, xₙ
- a complete assignment of values to every variable in the network
- parents(Xᵢ)
- the values, within that assignment, of the parents of node Xᵢ
- P(xᵢ | parents(Xᵢ))
- the entry of node Xᵢ’s conditional probability table for that row
- Πᵢ
- product over all n nodes; the factorisation encodes the conditional independences
Example of Bayesian Network
In the burglary network, the query "both neighbours have called; was there a burglary?" is P(B | j, m) = α Σₑ Σₐ P(B) P(e) P(a | B, e) P(j | a) P(m | a). Summing the four terms for each value of B gives the unnormalised pair ⟨0.00059224, 0.00149186⟩.
Normalising yields ⟨0.284, 0.716⟩: two independent reports raise a one-in-a-thousand prior to about 28 percent, and no higher, because the no-burglary mass of 0.00149186 arrives by three comparable routes - a causeless alarm then both calls 0.000628, no alarm but both calls anyway 0.000498, an earthquake alarm then both calls 0.000365 - which together outweigh the burglary route 0.000592 by 2.5 to one.
Variable elimination reaches the same answer by summing out Alarm first into a factor over (B, E), then Earthquake into a factor over B alone, so that the products at the leaves are computed once rather than once per branch of the enumeration tree.
Advantages and Disadvantages
Pros
- Represents a joint distribution over many variables with a number of parameters that grows with local structure rather than exponentially.
- Makes conditional-independence assumptions explicit and readable from the graph.
- Supports exact inference efficiently when the graph is a polytree, and consistent approximate inference otherwise.
Cons
- Exact inference is #P-hard in general, and sampling methods can converge slowly when evidence is improbable.
- The tables must be specified or learned, and their size grows exponentially with the number of parents.
- The graph depends on the order in which variables are introduced; a poor order gives a needlessly dense network.
Frequently Asked Questions
Do the arrows have to mean causation?
No. The arrows assert conditional dependence, and any ordering that places parents before children gives a valid network. Causal orderings are preferred in practice because they produce sparser graphs and tables whose entries people can actually assess.
What is a Markov blanket and why does it matter?
The Markov blanket of a node is its parents, its children, and its children’s other parents. Given the values of the blanket, the node is independent of every other variable in the network. Gibbs sampling relies on this: resampling one variable needs only a handful of table entries, never the whole joint.
When should I sample instead of computing exactly?
When the graph is multiply connected and variable elimination would produce factors too large to store. Rejection sampling is simplest but discards most samples once there are several evidence variables; likelihood weighting keeps every sample by fixing the evidence and weighting; Gibbs sampling walks a Markov chain whose stationary distribution is the posterior.
The Bottom Line
A Bayesian network is a joint distribution you can actually write down: a graph of direct influences plus small local tables, whose product is the joint and whose missing edges are the independence assumptions. Inference is a sum of products, cheap on polytrees and hard in general, which is where sampling takes over.