Our trade portal runs on six application servers behind a pair of HAProxy machines, and Ansible writes the HAProxy configuration. The template loops over the app group and takes each server's address from its facts:
server {{ host }} {{ hostvars[host]['ansible_default_ipv4']['address'] }}:8080 check
To make playbook runs faster, we turned on fact caching: gathering = smart and a jsonfile cache on the control node. With smart gathering Ansible collects facts from a host only when the cache has none for it, and the cache keeps them for fact_caching_timeout, which defaults to a day.
On a Wednesday in October we replaced two of the application servers with new instances built from a new image. They came up with new private addresses, the site playbook configured them, and they passed their checks. At half past eleven an engineer ran the load balancer playbook to change a timeout, limited to the lb group as the runbook says.
That play never contacts the application servers, so their hostvars came from the cache. For four of them the cached addresses were still right. For the two new ones the cache held facts gathered the previous morning, from the instances they had replaced. The template rendered, haproxy -c said the file was valid, the reload succeeded, and HAProxy's health checks marked two backends down. A third of our capacity disappeared just before the lunchtime peak, and the portal was slow for fifty minutes until somebody opened the stats page.
What hurt was what the cache had hidden. Without it, the application servers would have had no ansible_default_ipv4 in a play that never reached them, and the template would have failed on an undefined variable before touching a file. The cache turned a loud failure into a confident wrong answer.
The template now uses the private address that the aws_ec2 inventory plugin reads from the cloud API at the start of every run, and the inventory cache is off. After every reload the playbook reads the HAProxy stats socket and fails if any backend is down, so a bad address stops the play instead of the lunchtime trade. Replacing a server now runs the load balancer playbook as its last step, and the fact cache timeout is an hour.
A cache of facts is a set of claims about machines, and nothing in it expires when the machine does.
– Sergey Shinder
Top comments (0)