FEATUREDApplication Development & Modernization
PETER WELCHER | Solutions Architect
This blog is part of a series covering various SD-Access topics.
Previous blogs in this series include:
This blog covers planning IP Pools for SD-Access, specifically planning them across multiple sites.
If you’re wondering, “what’s the big deal, it’s just IP addressing” – well, that’s why I wrote this blog!
And yes, Cisco’s deployment documentation covers the How-To part of IP Pools quite adequately. You can create multiple global pools and then carve them up per-site for different purposes (data, wireless, etc.).
When you are fabric provisioning switches from a site, you supply IP pools by name as called for in dialog boxes. It is helpful to create and give suitable names to all the IP pools to assist in the later match-up process, following a naming convention. See below for more about this.
What Cisco’s documentation doesn’t do (as far as I can see) is provide a list of all the possible IP pools CCC (Cisco Catalyst Center) might use in one place, nor a discussion of how big they should be. With such a list in hand, one might then step back and plan addressing from that list. With one site, you can pretty much make it up as you go.
With 10, 20, 50 sites, that’s not such a good idea.
The intent is, if nothing else, to get you thinking about this topic if it is relevant to your present or future SD-Access design.
Naming Pools
Site-VN name (or purpose) is a good naming convention for IP pools. Example: NYC-STAFFVN. For a pool that is not per-VN, an example might be NYC-FABRICAPs. Or NYC-FABRIC-APS. There’s no charge for extra hyphens!
I am allergic to mixing hyphens and underscores in names; it creates potential errors. In fact, I’m a big fan of just removing underscore from keyboards. We do need some neutral separator here for readability.
You will probably want to establish a case convention: all upper, all lower, or something, to avoid configuration mismatches.
A Technicality
Technically, you create one or more global pools in CCC, and you then create the name and reserve sub-pools for various uses. This blog will just refer to IP pools, with the understanding that we’re focused on the sub-pools that get reserved.
A Disclaimer
Disclaimer: I personally have always really liked summarizable addressing, with ONE summarizable block per Site. That bias may permeate this blog!
Why summarizable? That way, you have some location information for troubleshooting. It is handy for the address to tell you something such as the site, without having to consult a per-/24 address block lookup spreadsheet. Or search in IPAM because prior assignments were a random mess.
Reality check: I sometimes see summarizable addressing in the field. There seems to be a lot more addressing that is anti-summarizable (if that’s a Thing). I’ve seen highly fragmented use of the address space, with /24’s handed out sequentially across sites, tracked in IPAM or a spreadsheet. Maybe that started historically and nobody has had the opportunity or desire to change it. (See also, “technical debt.”)
There’s no way you can efficiently do lookups with a small per-site XLS printout of per-site address blocks when each Site’s/24’s (or whatever size blocks) are essentially random. Hopefully, your IPAM tool facilitates IP lookups and has good descriptions of what each IP block is for. I’m not sure that is a realistic hope, based on my small sample size of what I’ve seen in the field, however.
The real cost here is that user and security analysis done subsequently involves a bit more effort to figure out what/where the endpoint(s) are compared to, say, actually recognizing some addresses (main campus, datacenter, etc.).
Let’s carry that rant a little further and then change the subject.
I’ve had success with campus or other equipment replacement using a repeatable addressing pattern. Things like, each building gets a block of some size, within that floors are blocks of some other size, and per-floor the first /24 is VoIP, the second is WLAN, etc. That sort of thing, where a per-site pattern means you can quickly take an address and not only determine the Site or building, but the floor, purpose, or other useful info.
We might refer to the site-first approach as “site. VN”. As in 10.site.VN.x if that fits your needs. Caution: you can blow through the entire 10.0.0.0/8 address block in a hurry that way! So, most sites can’t do that, attractive as it might look.
Let’s see (below) if I can persuade you that there’s something useful lurking here!
Summarizability by VN
My current thinking is that you might want to think about doing “VN.site” as your overall addressing approach.
If you can summarize the addressing by VN, that lets you write IP ACL-based micro-segmentation policy on a firewall or other device that does not do pxGrid / SXP. That is, a firewall external to the campus fabric(s), like an edge or datacenter entry firewall. Since SG’s in a VN are all in the same subnet, you’d lose micro-segmentation. But the macro-segmentation might be all you really need with well-chosen VN’s.
Here, I’ve just surfaced one of the security planning challenges relating to addressing: if you have firewalls or security devices that require address-based policy, because they don’t deal in SGT’s (Security Group Tags), then logical addressing becomes a much higher priority!
Recall my previous design advice: be stingy with VN’s, as they currently require extra effort and manual “glue” configuration on fusion firewalls or routers.
- What IP Pools Will CCC Want?
The following is my current list of IP pools I’ve encountered in SD-Access / Cisco Catalyst Center lab work. It attempts to be comprehensive. No guarantees! Once this is published, no doubt the next release of CCC will add more pools – Murphy’s Law. And note, there may be some I have not yet encountered in the CCC GUI, for whatever reason.
The IP pools come in two forms: per-VN and site-wide.
Per-VN pools:
- Users and/or devices (laptop, up to N personal devices, desk IP phone, or other networked devices. Wired or wireless. Printers, etc. What else?)
- Wireless users and devices, if using different addressing than wired addressing. I personally intend to keep it simple since SDA lets us treat wireless very much like wired.
- IP multicast loopback pool.
- (Spare per-VN pools. Because requirements will change over time.)
Site-wide pools:
- LAN Automation
- Fabric AP’s
- Border pool for BN’s
- Infra_VN pool
- (Spare pools for future needs)
Comments:
- It is rather inefficient to assume all VN pools are the same size. That way, you end up multiplying your worst case by the number of VN’s. I started with that for simplicity. It soon ate most of network 10. That’s a non-starter!
- Because sites will likely vary in the mix of users and devices across VN’s, do you really want to consider adjusting VN pool sizes to site-specific granularity? That sounds labor-intensive! Plus, if you get it wrong, it could get complicated.
- What I did in one site to reduce wasteful address space consumption was go with two sets of pools: one for “large” sites, and the other for “small”. The drawback to that is ending up with something like 10 overall address pools, each sub-divided in different ways.
The multicast loopback pool could be small; it needs to allow 1 per VN per device, as far as I can tell.
LAN Automation and IP Pools
CCC “LAN Automation” provides a way to discover and manage devices in CCC while provisioning them with standardized underlay connectivity.
The minimal IP pool for LAN automation is a /25. The pool should be “significantly larger” than the number of devices to be discovered. The devices to be discovered must all be at one Site. Since CDP is used, I presume there should not be crosslinks to another site.
You will need routing to this pool from CCC (and back). Of course, the pool must be unique.
For LAN Automation, you give it a global pool to use. It then automatically sub-divides it and uses it for three things:
- Pool for a temporary DHCP server for devices
- Pool for underlay link addressing
- Pool for assigning a loopback per device
See the LAN Automation Deployment Guide link under References below for more details of this, plus other information. Including how to do LAN Automation to discover and manage devices relatively quickly.
The Fine Manual (as in RTFM) says you can re-use a pool for more than one LAN Automation discovery. In that case, the pool should be at least a /24. I do not think this is a good idea. The point is that the LAN Automation pool is used for temporary DHCP addressing plus permanent addresses. When the LAN Automation is complete, the devices will be using some of the addresses permanently. The DHCP block will not be in use but cannot be reclaimed.
Links receive /30 addressing, so 4 IP addresses are consumed per link. Devices also get assigned a loopback. Another LAN automation sub-block is used for temporary DHCP addressing. You need to factor all this in when sizing the LAN automation pool.
If you plan on doing LAN Automation, you might add this to the IP pool spreadsheet discussed below. Note that you might want the LAN Automation pool to be chosen from a different address block, like 172.16.0.0/16. It can be convenient to have a clear distinction between underlay and overlay addressing.
In particular, if you’re doing SDA Transit, you can summarize both at the datacenter BN’s. The overlay just needs to know to go to the datacenter BNs to get to the underlay, and vice versa. In fact, the default route is all the sites overlays generally need to have from the datacenter BN’s. In effect, the underlay is “out there with the rest of the world.”
This alternative means having a pool for all underlay addresses and then assigning (reserving) per-site blocks out of that global pool. That sounds to me like another spreadsheet tab and then IPAM entries, tracking which underlay pool is at which site.
This is where having a regular topology at sites can help. If you have 1 BN or 2 BN’s and all the access switches attach to them, then the calculation is simple. Don’t forget to allow for a crosslink connecting two BNs, if that’s your design.
Thus for 1 BN, you would have N links to access switches, where N is how many there are.
For 2 BN’s, you would have 2 x N + 1: two links to each access switch, plus one crosslink between the BN’s.
A Calculating Approach
I wanted to get my head around the addressing implications and the likely network 10 consumption. So I built a spreadsheet. I’m going to describe it roughly so as to not propagate any bad assumptions or spreadsheet formula errors.
The spreadsheet works in terms of the number /24 blocks needed.
Input: approximate number of users at the Site. (For site-wide VN’s, anyway.)
Per VN:
- Calculate 4.25 times that, divided by approximately 250 usable addresses per /24, and round up. The 4 is for the user, 2 personal devices, and a desk phone. The .25 is for printers and shared devices. That gives the number of users/device /24 blocks per VN. Alternatively, use the number of active ports, and adjust if moving to IP phones, merging wired / WLAN, or otherwise increasing the number of addresses needed.
- If you later (or upfront) create a VN for IoT devices, is that going to be enough addresses? Something to consider, hard to predict. IOT could be a lot of devices.
- I don’t feel the need to address WLAN users separately, and it just complicates the calculations. (Assuming WLAN is being done SDA-style.) So, I assumed wireless users are the same as users on switch ports. No /24 blocks needed. It does mean the per-VN address pool(s) has to be bigger to accommodate wired + WLAN users.
- The IP multicast loopback pool uses one per VN per switch, so one /24 per VN is probably enough unless the site is large. That’s a /24 per VN across the entire site. Spare pools per VN, I added enough /24’s to add up to the next power of 2, since that summarizes well. Unless that gets ridiculously large. YMMV. It’s your planning spreadsheet!
- Add it all up, multiply by the number of VN’s and spare VN’s (which I also rounded up to the nearest power of 2). This is Subtotal #1.
Overall, not per-VN:
- LAN Automation. See above.
- Fabric AP addressing: 1 /24 unless there are more than 250 AP’s in a large site. They all get addresses, one per AP.
- Border pool: addressing for the BN’s. You’ll use a /30 for each VN, tunnels for SDA Transit addressing, etc. One /24 is probably plenty.
- INFRA_VN addressing: 1 /24.
- Spare site-wide pools: add enough to hit power of two.
- Add them all up. This is Subtotal #2.
To finish, add the two subtotals.
For example, for 125 or 250 users per site, I ended up with 72 x /24’s needed for a planned deployment. Rounding up to 128, that’s a /17, which is a lot. Squeezing that down a little gets you to 64. For bigger, I end up with a /16 per Site.
The major contributing factor is that you’re leaving room for growth in the number of VN’s, plus how many /24’s per VN. In my case, we plan 3 VN’s for now, with 5 for growth. That leads to 5 VN’s time 8 /24’s per VN or 40 /24’s consumed right there, not being immediately used.
More recently, I changed over to the “VN.site” approach. I took one big block and carved it up for VN.site, then about five smaller blocks for the other pools. That works out pretty cleanly (and more simply) but it sure consumes a lot of address space!
To deal with address sprawl, I / we then took that approach and did it for two sizes, with /20 and /22 per-VN addressing depending on site size. That helped with address consumption.
At the time I did this, it was a work in progress, and there was a pending discussion about maybe doing /21 and /23 (to reduce wasting address space due to many small sites of around 64 users).
Summarizability
The other major contributing factor is rounding up to a power of two, to carve up the address space clean, since ultimately, we’re taking things down to the bit level. That can consume a lot of address space.
In CCC, IP pools must be summarizable blocks. You do not actually put site pools in. But you maybe don’t want to just grab, say, the next 72 /24’s in a row, because the sub-blocks may not be summarizable. That’s why I put in rounding up.
One alternative would be to allocate just what you need right now, with a little padding to powers of two. And plan on doing similarly when more VN’s drive the need for more IP pools. The drawback here is, you won’t have one tidy IP block per Site.
There are other alternatives, like carving up a couple of big blocks by sites, e.g., one block for the per-VN stuff and one for the per-site stuff. But simple address summarization goes out the window with that.
To sum up, it’s not obvious how to do this cleanly. There are (as aways) trade-offs. The above provides a fair amount of room for growth but cannot possibly anticipate all future needs. And ultimately, you have to find an approach that works well for you and your site.
Note: If you err on the low side and need more addresses for a VN, you can add more pools to it. From my perspective, it would be preferable for these to come from spare space within a site block.
Update
Allocating ¼ or ½ of 10.0.0.0/8 just did not seem wise.
We decided to allocate per-VN blocks “just in time” for two sizes, large and small. That allows for learning/changes as we go, plus new requirements. For instance, fabric site-wide user VN’s require a user count suitably scaled up, but one or more IOT VN’s might have many more devices per Site, requiring larger blocks.
So, we’re starting out with 4 blocks for the initial 4 VN’s, with one perhaps being a /24 per Site for guests (actually a couple of different guest VNs for sites belonging one of two management/financial groupings for <<reasons>>, knowing that no site will be both flavors). The other VN’s are user-based, so /20 or /22 covers approx. 4000 or 1000 users, phones, etc., in the biggest VN at any site.
Anything bigger will have to be handled as a likely single exception, rather than burning up address space for a block to cover “very large” sites.
I hope there’s a hint there at an approach you might use. YMMV, of course.
Next Steps
The next step is to analyze the sites where SDA is to be deployed and classify them as to size.
That’s the other reason I built a spreadsheet. Once I got the formulas and layout, I copied the first tab and plugged in different numbers of users to see what resulted as far as /24 consumption.
I am currently working with a customer-provided spreadsheet, with per-site info: # of closets, biggest current closet switch stack and the number of active ports, the total number of active ports at the Site, and total standalone switches or stacks (devices needing addressing). AP’s we’ll deal with separately.
It’s a lot of work gathering accurate data for that! That’s planning for you!
It turned out that the data fell into 2 or 3 size “buckets.” We could then label each Site as to size and start assigning small and large blocks of addresses, whatever size those turn out to be based on the per-VN per-site spreadsheet for IP pools for that size.
Replacing all the equipment at once wasn’t going to happen (budget, staffing). That offered the opportunity to do mid-course corrections.
Other considerations:
- Having a good site by site inventory of switches and ports helps.
- Rule #1 in Excel is never aggregate cells, treat it like a flat database, one row per site/closet/switch capturing number of ports and their speeds. That facilitates rolling up data by site. Color code all you want, but don’t merge cells.
- With COVID, figuring out real port usage got harder, since people were doing WFH and the office switch ports are inactive. The flip side of that: having people absent from the site does help when doing equipment replacement and migration to SDA. Institutional knowledge helps!
- Outdoor wireless for large numbers (for distanced gatherings) is another potential special case.
Conclusion
I hope this discussion proves useful in your SD-Access address planning!
References
