Our inventory has two kinds of group. Role groups say what a server does: web, reporting, batch. Site groups say where it is: site_croydon, site_leeds. Both kinds sit directly under all. Site variables are mostly network things, the local DNS and NTP servers, but when we built Leeds as a home for small utility VMs somebody also put a sensible default there:
jvm_heap: 2g
The reporting group has its own value, because month end reports load a lot into memory:
jvm_heap: 12g
In September we moved the three reporting servers from Croydon to Leeds, which in the inventory meant moving them from one site group to another. The playbook ran clean. On the first working day of October every large month end report failed with OutOfMemoryError, and the JVM command line on each server said -Xmx2g.
When a host belongs to two groups, Ansible merges their variables in an order. A child group overrides its parent. Groups at the same level, which ours were, are merged alphabetically, and the group loaded last wins. Reporting sorts before site_leeds, so the site value overwrote the role value. In Croydon the same thing had been true, but site_croydon had never defined jvm_heap, so there was nothing to overwrite. Nobody had decided that sites outrank roles. The alphabet had decided it, and only for the variables both groups happened to set.
We took jvm_heap out of every site group, so site groups now hold only facts about the site. The reporting group sets ansible_group_priority in the inventory file itself, because that variable is read while groups are being loaded and is ignored in group_vars. And every pull request to the inventory runs ansible-inventory --list on the old and new versions and posts the difference in resolved variables per host. Moving three servers between sites now shows, before merge, that their heap would drop from twelve gigabytes to two.
A variable defined in two places has a winner whether or not anyone chose one. We now make sure someone has, and we look at the answer for every host before Ansible acts on it.
– Sergey Shinder
Top comments (0)