They clearly haven't talked to a telco or network device vendor, they would've sold them a VRF/EVPN/L3VPN based solution… for a whole bunch of money :)
You can DIY that these days though, plain Linux software stack, with optional hardware offload on some specific things and devices. Basically, you have a traffic distinguisher (VXLAN tunnel, MPLS label, SRv6, heck even GRE tunnel), keep a whole bunch of VRFs (man ip-vrf) around, and have your end services (server side) bind into appropriate VRFs as needed.
Also, yeah, with IPv6 you wouldn't have this problem. Regardless of whether it's GUAs or ULAs.
Also-also, you can do IPv6 on the server side until the NAT (which is in the same place as in the article), and have that NAT be a NAT64 with distinct IPv6 prefixes for each customer.
I like to think this is what we did. It's a simple Linux software stack - Linux, nftables, WireGuard, Go... But the goal was also to make it automatic and easy to use. It's not for my Mom. But you don't need a CCNP either.
The trick is in the automation and not the stack itself.
The key distinction with a L3VPN setup is that the packets are unmodified from and including the IP layer upwards, they're just encapsulated/labelled/tagged (depending on your choice of distinguisher). That encapsulation/… is a stateless operation, but comes at the cost of MTU (which in your case should be a controllable factor since the inner flows don't really hit uncontrolled devices.) Depending on what you're trying to do, the statelessness can be anything from useless to service critical (the latter if you're under some risk of DoS due to excessive state creation). It can also alleviate NAT problems, e.g. SIP and RTP are "annoying" to NAT.
(ed.: To be fair, 1:1 NAT can be almost stateless too, that is if your server side ["Technician"] can be 1:1 mapped into the customer's network, i.e. the other direction. This only works if you have very few devices on "your" side and/or/according to how many IPs you can grab on the customer network.)
The IPv6/NAT64 approach meanwhile is very similar to what you did, it just gets rid of the need to allocate unique IP addresses to devices. The first 96 bits of the IPv6 address become a customer/site ID, the last 32 bit are the unmodified device IPv4 address.
10. is /8 (24 payload bits), 172.16 is /12 (so 22) and 192.168 is /16. Very little need to spend more than 18 bits of space to map every 'usable' private IPv4 address once per customer. Probably also less than 14 bits (16k) of customers to service.
There's more addresses I didn't know about offhand but found when looking up the 'no DHCP server' autoconf IP address range (Link Local IPv4).
The problem with talking to a telco, is you have to talk with not just one but any your customer may use. And if at the customer location there’s multiple routers in between the cameras and that telco router, it’s a shitshow trying to configure anything.
Much easier to drop some router on site that is telco neutral and connect back to your telco neutral dc/hq.
No good when the upstream is some wifi connection provided by the building management, rather than a telco themselves.
May as well pick a single solution that works across all Internet connections and weird setups, be an expert in that, vs having to manage varying network approaches based on telco presence, local network equipment, operating country, etc.
On the later parts, VRF in my scenarios won’t scale.
Need to provide support access to 10k-50k locations all with the same subnet (industry standard equipment where the vendor mandates specific IP addressing, for better or worse). They are always feeding in data into the core too.
That is a valid point. Though I would probably check first what the scaling limits on VRFs actually are; there was some netdev work a while back to fix scaling with 100k to 1M devices (a VRF is a device, though also a bit more than that). It's only the server ("technician") that needs to have all of these (depends on the setup if that helps or not), intermediate devices just need to forward without looking at the tags, and the VPN entry point only cares about its own subset of customers.
I'd probably use the IPv6 + NAT64 setup in your situation.
What we could do is increase the number of IP addresses available. Just imagine if we enlarged the IP address space from 32 bits to 128 bits: Every device on the Internet could have a unique IP address!
That sounds apocalyptic. What if street addresses were unambiguous? Think of the security implications. Anyone could just walk into your house. Much better to just have "local street 10 b" etc.
The thing is, this upgrade you two are praising is designed to satisfy the original article's needs and no one else's.
Why do all those devices need to talk to each other btw? It's never specified. Is it a user need or a data collection/spyware need?
In a world where security articles make the news saying that you could obtain access to something IF the attacker already has local root and IF the moon is in a quarter phase and IF the attacker is physically present in the same room as the machine and this means the sky is falling...
... we should be questioning why disparate devices on unrelated home networks need to talk to each other.
Peer-to-peer requires that devices from different home networks talk to each other. Gaming, audio/video chat, screen sharing, file sharing (torrents), etc.
The whole idea of the internet from the beginning is that devices can talk with each other.
The need is real. You are a service provider. You need to manage equipment at customer sites. You need to access them simultaneously. But all the customers are using the same subnet...
If Bell gave out cellphones with the same phone number, how can you call anybody? But they still do.
Many devices have cloud access, but every manufacturer is different. It is a nightmare at scale.
The issue is that we DO NOT want every device to have a publicly routable IP address. It does make sense for some machines, but you probably don't want your your Internet-of-Shit devices to have public IPs. Of course you can firewall the devices, but you are always one misconfiguration or bug away from exposing devices that should not be exposed, when a local network is a more natural solution for what is supposed to remain local in the first place.
We did. It's called IPv6. It's 20 years old and still not usable universally. At the high end, like enterprise or telcos, it's fantastic. But at the grass roots level of residential and small businesses, it's still a nightmare.
We implemented a very similar solution more than five years ago. The NanoPi R3S was not available then, so we used the GL.iNet GL-MT300N-v2 (aka Mango) running OpenWRT as our edge gateways. It's slow and only has two 100Mb ports, but that was never the bottleneck. At that time, I was able to assemble a batch of 10 including cables and power supplies for only $300, which was ridiculously cheap for such a flexible solution.
If you need a polished, turnkey solution, by all means check netrinos out. If you have a strong Linux/nftables/wireguard background, this solution is easy to roll on your own.
I wouldn't be surprised if a lot of the hardware under management (e.g. IP cameras, NVRs, cable modems) lacks support for IPv6, and/or the customer networks that it's resident on don't have working IPv6 transit.
The solution is to run ipv6 on the overlay and have the customer site gateway thing they have to translate it to target ipv4. Conveniently you can do the translation it more or less statefully and very easily because you can just embed the ipv4 addr in ipv6. For example you could grab a /64 prefix, assign 32 bits to customer/gateway id and other 32 bits to target ipv4 addr.
Coincidentally I think that's an overestimation on the number of devices that don't support IPv6. At this point, vendors have to go out of their way to disable IPv6, and they lose out on some government/enterprise tenders that require IPv6 even if they're not running it (yet).
IPv6 is very badly supported at the low end of the market. Cheap webcams, doorbells, etc. And that not counting already old equipment...
If we had a nuclear war, we could start over. But for now, we are stuck. Blame it on Cisco for inventing NAT.
man this sounds so realistic, a system administrator saying "yes! Now we can migrate to the IPv6" after the 99.9 percent of the world population is killed
Yes, I was going to suggest nat64 encapsulating the customer's v4 network on the wireguard overlay, but their embedded device is presumably a little linux board, and mainline linux still lacks any siit/clat/nat64 in netfilter. So I guess they'd end up in a world of pain with out-of-tree modules like jool or inefficient funnelling through taptun tayga-style.
IPv6 solves the addressing problem, not the reachability problem. Good luck opening ports in the stateful IPv6 firewalls in the scenarios outlined in TFA:
> And that assumes a single NAT. Many sites have a security firewall behind the ISP modem, or a cellular modem in front of it. Double or triple NAT means configuring port forwarding on two or three devices in series, any of which can be reset or replaced independently.
I'm not really seeing a reason why it would be impossible to open firewalls in that scenario. More work, sure, but by no means impossible. In any case TFA says right up front that it is trying to solve the problem of overlapping subnets, which IPv6 solves nicely.
Then you've probably never worked in any serious networked embedded systems space. Getting people to open ports on the firewall and making the firewall configuration palatable to the end customer is like a quarter of what I think about when my team makes new features.
Hole punching actually works most of the time. A lot more often than you might think. But enterprise firewalls usually don't allow it. And some home routers fail when you check all the anti-intrusion options. But it's the same for other VPNs.
In the residential and small-business space, it's pretty rare. You might need to point it out to the network guy. If the customer wants the service, they should be open to it.
The problem isn’t that it doesn’t work (and it does often not work – one “symmetric NAT” in the old/deprecated terminology is enough), it’s that it’s orders of magnitude more complex than it needs to be.
I’ve also never seen it work for TCP in practice, and not everybody should have to roll their own UDP wrapper for their TCP-expecting application.
Hole punching is a thing. Ports are not normally completely blocked. They allow replies, which can be exploited to do make a connection. Obviously this requires an out of band signaling mechanism. Tailscale does this, so does WebRTC, iirc.
Yes, but I don't believe all firewalls support that, especially for TCP, and as you've mentioned, now you also need to maintain a handshaking mechanism.
The complexity makes sense if you need to transport a lot of data peer-to-peer or the lowest possible latency, but if you don't, you might as well use that coordination server (which outbound-only clients are connecting to) for payload communication as well.
Firewalls don't track connections as carefully as you might think. They don't delete the mapping when an RST is received, so you can keep using it and receive a SYN in the other direction and now you have a connection.
Companies with an IT department, maybe. Companies without IT, not much. People, nope.
I can't see my neighbors opening ports on their switch. What's a switch, to start with. And what happens when they change provider and switch next month?
It's much easier to tell them: I install two boxes. One is the camera (or whatever), the other one is necessary to make the camera work properly, keep it online, don't switch it off.
> I'm not really seeing a reason why it would be impossible to open firewalls in that scenario.
Cheap ass ISP-managed routers. Got to be lucky for these rubbish bins to even somewhat reliably provide IPv6 connectivity to clients at all, or you run into bullshit like new /64's being assigned every 24 hours, or they may provide IPv6 but not provide any firewall control...
You can have /both/ a ULA and a Globally Routable address. In practice it works just fine. My internal DNS points to the ULA for internal connectivity and my hosts use their global addresses for external connectivity.
Ah, you mean for cases where you want both stable addresses (even if only internal) and globally reachable ones (even if non-constant)?
Yeah, that works, but everything gets much easier if your internal DNS can just support the varying prefix natively, e.g. via integration with the external-facing DHCP or PPPoE or whatever other address configuration protocol you use, since then you can reach everything both locally and globally by name.
It also gets more fragile. If your ISP can't or doesn't issue you a prefix for whatever reason then your entire IPv6 network stops working even internally. This is even more pertinent if, like me, you're on a 4G LTE connection. Verizon has great IPv6 support, when you can get it, and when you can't I'd still prefer to have a stable internal network.
That's the addressing problem, although I have some bad news on that: NAT is used with IPv6 in some places.
The reachability problem is, even with public addresses, sometimes you have to do the same thing to "configure port forwarding" with stateful IPv6 firewalls as with double or triple NAT IPv4.
I recently just changed my default subnet to 10.X.Y.... rolling two random numbers to make it highly unlikely my home subnet through wireguard would conflict with the subnet where I am connecting from.
My (very large) corporate network uses 172.16 and 10. heavily, which has lead me to set my docker/daemon.json default-address-pools to 84.54.64.0/18, as it's very unlikely we need to communicate with any IPs in Uzbekistan.
When I separated my scientific instruments from IT, I went to fixed IP and set each device to 192.A.B.x where x is different for each instrument or PC. And A & B are for my lab only, but definitely not the same as the "generic" address range IT is using.
One day somebody working days or nights "helpfully" plugged one of IT's loose office-machine-network cables into one of my little lab ethernet switches which had a vacant spot :\
With separate IP subnets it really kept the traffic from crossing, no damage was done, and nobody ever knew until a PC configured for DHCP was plugged into the lab network, and their router wanted to autoassign an IP address to it.
C) I got teased for it a long time ago by my other nerd friends.
But the US DOD has huge blocks of prefixes that it doesn't do anything with, presumably they use it for internal routing so every device they have could publicly route without NAT..
One of those prefixes is 7.0.0.0/8.
My home network uses that. I have never had an issue with S2S VPNs.
However, there have been a few bits of software (pfsense for example) which have RFC1918 hardcoded in some areas and treat it like a public network and overwriting it means doing the entire network setup manually without the helping hand of the system to build-out a working boilerplate.
In this vein there's also 3 TEST-NETs, all /24 but still useful. I've been known to use TEST-NET 1 for Wireguard: 192.0.2.0/24. The other two are 198.51.100.0/24 and 203.0.113.0/24.
There's also 198.18.0.0/15, Wikipedia says it's "Used for benchmark testing of inter-network communications between two separate subnets"[1]. Use this if you really want to thumb your nose at the RFC police.
Do you run Docker? Because I remember having to VPN out to a client that used that range, and it caused conflicts where our docker containers couldn't reach the client side to fetch data.
We chose Go as the development language. Go produces statically compiled binaries that include all dependencies. The only external deps are wireguard, nftables, nmap, etc. All easy stuff. So we have no need for Docker. We publish binaries for ARM64 and AMD64. Avoiding Docker has made it much easier to work with.
I had this happen at home. I'm not convinced it was a good idea to choose default subnets as /20.
It was pretty easy to cause myself problems with Docker compose. Eventually I run out of subnets in the 172.16 range and it happily created subnets in the 192.168. range. Some of them overlapped with subnets on my LAN.
This works fine for your end. But the issue we are addressing is on the other end, when you don't control the network and need to reach devices. If all customer sites are running rfc-unroutable blocks, you eventually encounter conflicts. And the conflict will likely be with the 2nd one you try.
I mostly wireguard in from my work's guest wifi and people's homes. The first I don't have access to anything internal anyways and it doesn't conflict and the latter mostly use default 192.168.1.0/24 so there's no conflicts I've hit there so far.
The IETF really dragged their heels on CGNAT because they thought that IPv6 is easy™ (of course not, it's intentionally designed not to be "almost the same but wider" but include unworkable stuff like Mobile IPv6[1] which is just a fancy VPN) until they were forced to allocate 100.64.0.0/10 because some ISPs are not just using 10.0.0.0/8 but also US-DoD addresses (especially 11.0.0.0/8, because it's basically 10.0.0.0/7) as "private" addresses.
[1] Not IPv6 on mobile devices but a fully-owned IPv6 range that is supposed to be the address for a device regardless of where it is, see RFC 3775
Are those usually visible to clients sitting behind routers though? I'm not super familiar but the things I'm seeing make it seem like that should only be visible IPs on the internal network of carriers which is not a place I am ever connecting from.
This is basically what I use tailscale & their magicdns feature for. I manage a few locally hosted jellyfin servers for myself and some family members, and its the same problem. I just added tailscale to them all and now I can basically do ssh parents.jellyfin.ts.net or inlaws.jellyfin.ts.net
I need to implement this type of thing for supporting networks of family members, but without the media server aspect - just computer/networking support. I'm looking for a cheap and reliable device that I can put in each home, to give the Tailscale "foothold". Do you happen to know of any tiny devices? I was thinking there must be something even cheaper than a Raspberry Pi to perform this single function at each location.
If they have an Apple TV, you can just install the app and use it as an exit node. I would check out the devices that are on their network currently, chances are you can use one of those.
An old micro pc from dell/hp/lenovo. They are often cheaper and more capable than Raspberry Pis. You can just put up a random Linux distro and it will work.
The only drawback are routes - they won't work on the same CIDR (I mean the fact that you can say in Tailscale "if you want to reach the 192.168.16.13 device that does not support Tailscale, go through this Tailscale gateway"). For this I had to shift my parents' network to be able to access stuff like the printer, in a network that clashed with another one of mine.
The way we did it, roting is not a problem. Any Netrinos client (Windows, Mac, or Linux, including the free version) can act as a gateway. It assigns a unique overlay IP to devices on the local network that can't run software themselves, like cameras, NAS units, or printers, and handles the NAT translation.
Think of it like a router's DMZ feature, but inverted. Instead of exposing one device to the internet, each device gets a private address that's only reachable inside your mesh network.
In your experience, how often does Tailscale have to resort to an external relay server to traverse? I’ve had that out the kibosh on bandwidth/latency sensitive applications before.
> The gateway device performs 1:1 NAT. Traffic arriving for 100.97.14.3 is destination-translated to 192.168.1.100, and the source is masqueraded to the gateway's own LAN address.
Couldn't you tell the WG devices that 192.168.2.0/24 refers to the 192.168.1.0/24 network at customer A, such that 192.168.2.55 is routed to 192.168.1.55. Same for 192.168.3.0/24 referring to customer B.
I think this is what the article is getting at but I don't see the value in manually assigning an alias to each non-wg device, versus assigning an alias to the entire LAN.
The problem there is you still need to keep track of the subnets. It works for a while, but it's quite complex. NAT is actually easier when you get into hundreds of sites.
Yeah so instead DNAT, use NETMAP on the gateway device to that LAN. (Sorry if I'm abusing the terminology, I only do this stuff like once a year for homelab.)
eg this is what I'm currently using to alias my home network
The suggested solution involves using the CGNAT /10 in conjunction with a VPN, but I've actually seen someone do this, and still have problems with certain end users where their next hop for routing also involves a router with an IPv4 address in the same space, so it's not really bulletproof either. We may as well consider doing other naughty things like co-opting DoD non-routable /8s or the test net in the RFCs you're not supposed to use, because basically anything you pick is going to have problems.
That does not happen here. The CGNAT addresses are in the VPN tunnel. And the tunnel connects private devices end-to-end. The LAN packets never see the Internet. They are inside the WireGuard packets.
This is what the NETMAP target in iptables is for - map an entire subnet to another subnet, including the reverse. We were doing this 20 years ago for clients trying to on-board other companies that they'd bought. It's horrible, but it does solve the problem in a pinch.
I feel like this is really only an issue with true site to site VPNs. Client to site shouldn't have this issue because the VPN concentrator is like a virtual NAT.
The best strategy might be to maintain the ability to easily reassign the network for a site. If every site is non-overlapping the problem does become trivial. I'd much rather fight a one time "reboot your machines tonight" battle than the ongoing misery of mapping things that do not want to be.
One step beyond this is the multi-subnetted network on each side. You get the DNAT working, but then suddenly the app gets more complex over time and suddenly you're calling 192.168.2.x, which leads to async routes. Some traffic works, some traffic works one way, and other traffic disappears.
Then you as the client/app manager pull your hair out as the network team tells you everything is working fine.
Shameless plug - this is exactly the same problem that our team had when we had to maintain a bunch of our customer's servers. All of the subnets were same, and we had to jump through hoops just to access those servers - vpns, port forwarding, dynamic dns with vnc - we've tried it all. That is why we developed https://sshreach.me/ - now it's a click of a button.
The initial idea started as a bunch of ssh tunnels. Been doing that for years. But WireGuard seemed a better solution at scale, and more efficient. When I first saw WiteGuard, it blew my mind how elegantly simple it was. I always hated VPNs. Now I seem to have made them my life...
Your website landing page is great. No stock photo hipsters drinking coffee, no corporate fluff amid whitespace wasteland. Just straight to the point. Rare sight today.
I decided to learn IPv6 recently and I'm pleasantly surprised how simple and elegant it is. Truly a joy. Highly recommend, if you've never worked with IPv6 to try it. It's like discovering a bidet.
> But the moment two sites share the same address range, you have an ambiguity that IP routing cannot resolve.
Writing PF or nft rules to NAT these hyper-legacy subnets on the local side of the layer3 tunnel is actually super trivial, like 20 seconds of effort to reason about and write in a config manifest.
Like written the article, a device on the customer site is required. At that point you might as well deploy a router that has a supportable software stack and where possible sober IP instead of legacy IP.
.
I have been running IPv6-only networks since 2005 and have been deploying IPv6-only networks since 2009. When I encountered a small implementation gap in my favorite BSD, I wrote and submitted a patch.
Anyone who complained about their favorite open source OS having an IPv6 implementation gap or was using proprietary software (and then also dumb enough to complain about it), should be ashamed of themselves for doing so on any forum with "hacker" in the name. But we all know they aren't ashamed of themselves because the competency crisis is very real and the coddle culture let's such disease fester.
There is no excuse to not deploy at minimum a dual-stack network if not an IPv6-only network. If you deploy an IPv4-only network you are incompetent, you are shitting up the internet for everyone else, and it would be better for all of humanity if you kept any and all enthusiasm you have for computers entirely to yourself (not a single utterance).
I won't name the 2 large telecoms I know, that don't support IPv6 being used by customers - if you get L2VPN, L3VPN, other typical services etc. it will be IPv4-only. Of course you can buy a wave and do whatever you want with it :-)
Support for IPv6 is notoriously bad in residential modems. They can barely run IPv4. In an enterprise, you can do it properly. But here we are stuck with the junk the ISP gave out. Customers don't care. You have to work with what you've got.
>Support for IPv6 is notoriously bad in residential modems.
No? Over here at (South) East Asia we have been deploying IPv6 for nearly a decade now. The users are getting their IPv6 connectivity. Before someone jumps out and shouts SeCuRiTy: the firewall is enabled by default.
I am not saying the support is perfect. I know some people moan about lackluster IPv6 configuration in many routers. But for 90% of residential internet users (who care about pretty much nothing but the ability to watch YouTube and browsing social media), it damn sure is.
They clearly haven't talked to a telco or network device vendor, they would've sold them a VRF/EVPN/L3VPN based solution… for a whole bunch of money :)
You can DIY that these days though, plain Linux software stack, with optional hardware offload on some specific things and devices. Basically, you have a traffic distinguisher (VXLAN tunnel, MPLS label, SRv6, heck even GRE tunnel), keep a whole bunch of VRFs (man ip-vrf) around, and have your end services (server side) bind into appropriate VRFs as needed.
Also, yeah, with IPv6 you wouldn't have this problem. Regardless of whether it's GUAs or ULAs.
Also-also, you can do IPv6 on the server side until the NAT (which is in the same place as in the article), and have that NAT be a NAT64 with distinct IPv6 prefixes for each customer.
I like to think this is what we did. It's a simple Linux software stack - Linux, nftables, WireGuard, Go... But the goal was also to make it automatic and easy to use. It's not for my Mom. But you don't need a CCNP either. The trick is in the automation and not the stack itself.
The key distinction with a L3VPN setup is that the packets are unmodified from and including the IP layer upwards, they're just encapsulated/labelled/tagged (depending on your choice of distinguisher). That encapsulation/… is a stateless operation, but comes at the cost of MTU (which in your case should be a controllable factor since the inner flows don't really hit uncontrolled devices.) Depending on what you're trying to do, the statelessness can be anything from useless to service critical (the latter if you're under some risk of DoS due to excessive state creation). It can also alleviate NAT problems, e.g. SIP and RTP are "annoying" to NAT.
(ed.: To be fair, 1:1 NAT can be almost stateless too, that is if your server side ["Technician"] can be 1:1 mapped into the customer's network, i.e. the other direction. This only works if you have very few devices on "your" side and/or/according to how many IPs you can grab on the customer network.)
The IPv6/NAT64 approach meanwhile is very similar to what you did, it just gets rid of the need to allocate unique IP addresses to devices. The first 96 bits of the IPv6 address become a customer/site ID, the last 32 bit are the unmodified device IPv4 address.
10. is /8 (24 payload bits), 172.16 is /12 (so 22) and 192.168 is /16. Very little need to spend more than 18 bits of space to map every 'usable' private IPv4 address once per customer. Probably also less than 14 bits (16k) of customers to service.
There's more addresses I didn't know about offhand but found when looking up the 'no DHCP server' autoconf IP address range (Link Local IPv4).
https://en.wikipedia.org/wiki/IPv4#Special-use_addresses
That's all true on a statement level, but doesn't make an IPv4:IPv4 NAT solution better than either VRF/encap or IPv6 mapping.
The benefit with VRF/encap is that the IPv4 packets are unmodified.
The benefit with IPv6 mapping is that you don't need to manage IPv4:IPv4 tables and have a clear boundary of concerns & zoning.
In both cases you don't give a rat's ass which prefixes the customer uses. That math/estimation you're doing there… just entirely not needed.
The problem with talking to a telco, is you have to talk with not just one but any your customer may use. And if at the customer location there’s multiple routers in between the cameras and that telco router, it’s a shitshow trying to configure anything.
Much easier to drop some router on site that is telco neutral and connect back to your telco neutral dc/hq.
The Metro Ethernet Forum standardized a lot of services telcos can offer, many years ago
No good when the upstream is some wifi connection provided by the building management, rather than a telco themselves.
May as well pick a single solution that works across all Internet connections and weird setups, be an expert in that, vs having to manage varying network approaches based on telco presence, local network equipment, operating country, etc.
That's all true, but you can also, you know, like, talk to people without buying your whole solution from them :)
(btw, have you actually read past the first 7 words? I'm much more interested what people think about the latter parts.)
On the later parts, VRF in my scenarios won’t scale.
Need to provide support access to 10k-50k locations all with the same subnet (industry standard equipment where the vendor mandates specific IP addressing, for better or worse). They are always feeding in data into the core too.
Much easier to just VPN+NAT.
That is a valid point. Though I would probably check first what the scaling limits on VRFs actually are; there was some netdev work a while back to fix scaling with 100k to 1M devices (a VRF is a device, though also a bit more than that). It's only the server ("technician") that needs to have all of these (depends on the setup if that helps or not), intermediate devices just need to forward without looking at the tags, and the VPN entry point only cares about its own subset of customers.
I'd probably use the IPv6 + NAT64 setup in your situation.
What we could do is increase the number of IP addresses available. Just imagine if we enlarged the IP address space from 32 bits to 128 bits: Every device on the Internet could have a unique IP address!
That sounds apocalyptic. What if street addresses were unambiguous? Think of the security implications. Anyone could just walk into your house. Much better to just have "local street 10 b" etc.
You could install a door. Then again, who am I telling people what to do.
Interesting idea! But I think such upgrade would take years, if not decades, to get widely adopted.
Or maybe a century.
The thing is, this upgrade you two are praising is designed to satisfy the original article's needs and no one else's.
Why do all those devices need to talk to each other btw? It's never specified. Is it a user need or a data collection/spyware need?
In a world where security articles make the news saying that you could obtain access to something IF the attacker already has local root and IF the moon is in a quarter phase and IF the attacker is physically present in the same room as the machine and this means the sky is falling...
... we should be questioning why disparate devices on unrelated home networks need to talk to each other.
Peer-to-peer requires that devices from different home networks talk to each other. Gaming, audio/video chat, screen sharing, file sharing (torrents), etc.
The whole idea of the internet from the beginning is that devices can talk with each other.
The need is real. You are a service provider. You need to manage equipment at customer sites. You need to access them simultaneously. But all the customers are using the same subnet... If Bell gave out cellphones with the same phone number, how can you call anybody? But they still do. Many devices have cloud access, but every manufacturer is different. It is a nightmare at scale.
There are completely legitimate usecases that are not "spyware" related for true end-to-end connectivity
For security there is still the firewall
And we could represent the addresses with hex numbers separated by : instead of decimal numbers separated by .
That’d be kinda inconvenient with respect to the port number syntax in URLs, though.
I heard there's some people working on a system that allows you to use names, but it seems to be very poorly designed and cause of a lot of outages.
The issue is that we DO NOT want every device to have a publicly routable IP address. It does make sense for some machines, but you probably don't want your your Internet-of-Shit devices to have public IPs. Of course you can firewall the devices, but you are always one misconfiguration or bug away from exposing devices that should not be exposed, when a local network is a more natural solution for what is supposed to remain local in the first place.
- NAT is not a firewall. A firewall is a firewall. - NAT is not better than a firewall. - NAT does not replace a firewall.
We did. It's called IPv6. It's 20 years old and still not usable universally. At the high end, like enterprise or telcos, it's fantastic. But at the grass roots level of residential and small businesses, it's still a nightmare.
I think that was the joke
[dead]
We implemented a very similar solution more than five years ago. The NanoPi R3S was not available then, so we used the GL.iNet GL-MT300N-v2 (aka Mango) running OpenWRT as our edge gateways. It's slow and only has two 100Mb ports, but that was never the bottleneck. At that time, I was able to assemble a batch of 10 including cables and power supplies for only $300, which was ridiculously cheap for such a flexible solution. If you need a polished, turnkey solution, by all means check netrinos out. If you have a strong Linux/nftables/wireguard background, this solution is easy to roll on your own.
Why not IPv6? Pretending that it doesn't exist??
https://en.wikipedia.org/wiki/List_of_IPv6_transition_mechan...
I wouldn't be surprised if a lot of the hardware under management (e.g. IP cameras, NVRs, cable modems) lacks support for IPv6, and/or the customer networks that it's resident on don't have working IPv6 transit.
The solution is to run ipv6 on the overlay and have the customer site gateway thing they have to translate it to target ipv4. Conveniently you can do the translation it more or less statefully and very easily because you can just embed the ipv4 addr in ipv6. For example you could grab a /64 prefix, assign 32 bits to customer/gateway id and other 32 bits to target ipv4 addr.
Basically Teredo.
Basically NAT64. (Teredo requires the IPv4 hosts to have awareness of it, this doesn't.)
It’s definitely on the software side… The human side.
The squishy side.
Coincidentally I think that's an overestimation on the number of devices that don't support IPv6. At this point, vendors have to go out of their way to disable IPv6, and they lose out on some government/enterprise tenders that require IPv6 even if they're not running it (yet).
Right, IPv6 is baked into the NIC, so it’s up to developers to use it.
IPv6 is very badly supported at the low end of the market. Cheap webcams, doorbells, etc. And that not counting already old equipment... If we had a nuclear war, we could start over. But for now, we are stuck. Blame it on Cisco for inventing NAT.
man this sounds so realistic, a system administrator saying "yes! Now we can migrate to the IPv6" after the 99.9 percent of the world population is killed
Like city planners after WWII
Yes, I was going to suggest nat64 encapsulating the customer's v4 network on the wireguard overlay, but their embedded device is presumably a little linux board, and mainline linux still lacks any siit/clat/nat64 in netfilter. So I guess they'd end up in a world of pain with out-of-tree modules like jool or inefficient funnelling through taptun tayga-style.
IPv6 solves the addressing problem, not the reachability problem. Good luck opening ports in the stateful IPv6 firewalls in the scenarios outlined in TFA:
> And that assumes a single NAT. Many sites have a security firewall behind the ISP modem, or a cellular modem in front of it. Double or triple NAT means configuring port forwarding on two or three devices in series, any of which can be reset or replaced independently.
The article's proposed solution for IPv4 is a combination of VPN+NAT. The solution in IPv6 can be just VPN, sans NAT.
It can be sans NAT, but there's no guarantee it will be, and if a remote site chooses to NAT their IPV6 you still have to support it
I'm not really seeing a reason why it would be impossible to open firewalls in that scenario. More work, sure, but by no means impossible. In any case TFA says right up front that it is trying to solve the problem of overlapping subnets, which IPv6 solves nicely.
Then you've probably never worked in any serious networked embedded systems space. Getting people to open ports on the firewall and making the firewall configuration palatable to the end customer is like a quarter of what I think about when my team makes new features.
Yes! Exactly this.
It's completely impossible if you simply don't have the necessary access. Not everybody can administer all firewalls upstream from them.
Nor can everyone control whether their connection supports v6, unfortunately.
Hole punching actually works most of the time. A lot more often than you might think. But enterprise firewalls usually don't allow it. And some home routers fail when you check all the anti-intrusion options. But it's the same for other VPNs. In the residential and small-business space, it's pretty rare. You might need to point it out to the network guy. If the customer wants the service, they should be open to it.
The problem isn’t that it doesn’t work (and it does often not work – one “symmetric NAT” in the old/deprecated terminology is enough), it’s that it’s orders of magnitude more complex than it needs to be.
I’ve also never seen it work for TCP in practice, and not everybody should have to roll their own UDP wrapper for their TCP-expecting application.
Hole punching is a thing. Ports are not normally completely blocked. They allow replies, which can be exploited to do make a connection. Obviously this requires an out of band signaling mechanism. Tailscale does this, so does WebRTC, iirc.
See: https://tailscale.com/blog/how-nat-traversal-works
Yes, but I don't believe all firewalls support that, especially for TCP, and as you've mentioned, now you also need to maintain a handshaking mechanism.
The complexity makes sense if you need to transport a lot of data peer-to-peer or the lowest possible latency, but if you don't, you might as well use that coordination server (which outbound-only clients are connecting to) for payload communication as well.
Firewalls don't track connections as carefully as you might think. They don't delete the mapping when an RST is received, so you can keep using it and receive a SYN in the other direction and now you have a connection.
Companies with an IT department, maybe. Companies without IT, not much. People, nope.
I can't see my neighbors opening ports on their switch. What's a switch, to start with. And what happens when they change provider and switch next month?
It's much easier to tell them: I install two boxes. One is the camera (or whatever), the other one is necessary to make the camera work properly, keep it online, don't switch it off.
> I'm not really seeing a reason why it would be impossible to open firewalls in that scenario.
Cheap ass ISP-managed routers. Got to be lucky for these rubbish bins to even somewhat reliably provide IPv6 connectivity to clients at all, or you run into bullshit like new /64's being assigned every 24 hours, or they may provide IPv6 but not provide any firewall control...
> or you run into bullshit like new /64's being assigned every 24 hours
It'd be nice if DNS servers supported this. Save the 64 host bits in the zone and just use whatever 64 prefix bits happen to be issued right now.
Otherwise it makes a strong case for the continued use of "private networks" and the IPv6 ULA mechanism.
> Otherwise it makes a strong case for the continued use of "private networks" and the IPv6 ULA mechanism.
Let's please not. Even without inbound reachability, hole punching is significantly easier given globally routeable addresses.
You can have /both/ a ULA and a Globally Routable address. In practice it works just fine. My internal DNS points to the ULA for internal connectivity and my hosts use their global addresses for external connectivity.
Ah, you mean for cases where you want both stable addresses (even if only internal) and globally reachable ones (even if non-constant)?
Yeah, that works, but everything gets much easier if your internal DNS can just support the varying prefix natively, e.g. via integration with the external-facing DHCP or PPPoE or whatever other address configuration protocol you use, since then you can reach everything both locally and globally by name.
> but everything gets much easier
It also gets more fragile. If your ISP can't or doesn't issue you a prefix for whatever reason then your entire IPv6 network stops working even internally. This is even more pertinent if, like me, you're on a 4G LTE connection. Verizon has great IPv6 support, when you can get it, and when you can't I'd still prefer to have a stable internal network.
[dead]
With IPv6 you don’t forward ports at all. The device already has a public address.
That's why I said "open ports", not "forward ports".
Stateful firewalls are very much a thing on v6. Many mobile ISPs don't allow incoming connections by default, for example.
Many CPEs (home routers) also come with a v6 firewall (I'd guess it's probably more common than not?), and not everybody has admin access to theirs.
That's the addressing problem, although I have some bad news on that: NAT is used with IPv6 in some places.
The reachability problem is, even with public addresses, sometimes you have to do the same thing to "configure port forwarding" with stateful IPv6 firewalls as with double or triple NAT IPv4.
You don't have to forward ports with IPv6 but you certainly can, and that means some remote sites you have to interact with will.
I recently just changed my default subnet to 10.X.Y.... rolling two random numbers to make it highly unlikely my home subnet through wireguard would conflict with the subnet where I am connecting from.
I’ll never not use 10.0.0.0/24 for the sole benefit of being able to collapse the zero and go ping 10.1.
Plus, most network admins think of you and aren’t so bold as to use the first subnet in the range, so I never had problems yet :)
I just use /24s in the lower-middle range of 172.16. Very unlikely to have a conflict there.
My (very large) corporate network uses 172.16 and 10. heavily, which has lead me to set my docker/daemon.json default-address-pools to 84.54.64.0/18, as it's very unlikely we need to communicate with any IPs in Uzbekistan.
When I separated my scientific instruments from IT, I went to fixed IP and set each device to 192.A.B.x where x is different for each instrument or PC. And A & B are for my lab only, but definitely not the same as the "generic" address range IT is using.
One day somebody working days or nights "helpfully" plugged one of IT's loose office-machine-network cables into one of my little lab ethernet switches which had a vacant spot :\
With separate IP subnets it really kept the traffic from crossing, no damage was done, and nobody ever knew until a PC configured for DHCP was plugged into the lab network, and their router wanted to autoassign an IP address to it.
So, uh.
I kinda don't want to share this because:
A) it's a bad idea
B) it means it will be less unique
and
C) I got teased for it a long time ago by my other nerd friends.
But the US DOD has huge blocks of prefixes that it doesn't do anything with, presumably they use it for internal routing so every device they have could publicly route without NAT..
One of those prefixes is 7.0.0.0/8.
My home network uses that. I have never had an issue with S2S VPNs.
However, there have been a few bits of software (pfsense for example) which have RFC1918 hardcoded in some areas and treat it like a public network and overwriting it means doing the entire network setup manually without the helping hand of the system to build-out a working boilerplate.
In this vein there's also 3 TEST-NETs, all /24 but still useful. I've been known to use TEST-NET 1 for Wireguard: 192.0.2.0/24. The other two are 198.51.100.0/24 and 203.0.113.0/24.
There's also 198.18.0.0/15, Wikipedia says it's "Used for benchmark testing of inter-network communications between two separate subnets"[1]. Use this if you really want to thumb your nose at the RFC police.
[1] https://en.wikipedia.org/wiki/List_of_reserved_IP_addresses
I actually looked at using those before the CGNAT range, but many of those blocks have been returned to the public Internet.
Do you run Docker? Because I remember having to VPN out to a client that used that range, and it caused conflicts where our docker containers couldn't reach the client side to fetch data.
Docker defaults to 172.16.0.0/16.
We chose Go as the development language. Go produces statically compiled binaries that include all dependencies. The only external deps are wireguard, nftables, nmap, etc. All easy stuff. So we have no need for Docker. We publish binaries for ARM64 and AMD64. Avoiding Docker has made it much easier to work with.
I had this happen at home. I'm not convinced it was a good idea to choose default subnets as /20.
It was pretty easy to cause myself problems with Docker compose. Eventually I run out of subnets in the 172.16 range and it happily created subnets in the 192.168. range. Some of them overlapped with subnets on my LAN.
Yes, we use Docker (or podman) but generally never rely on Docker’s internal address ranges.
I often use 172.31/16 for subnets and have never seen a conflict. I have seen 172.24 and 172.16 used before, though.
I find a lot of Docker containers using subnets inside 172.16.0.0/16.
Probably for the same reason – 172.16/12 is not as widely used for other networks :-)
This works fine for your end. But the issue we are addressing is on the other end, when you don't control the network and need to reach devices. If all customer sites are running rfc-unroutable blocks, you eventually encounter conflicts. And the conflict will likely be with the 2nd one you try.
I mostly wireguard in from my work's guest wifi and people's homes. The first I don't have access to anything internal anyways and it doesn't conflict and the latter mostly use default 192.168.1.0/24 so there's no conflicts I've hit there so far.
I subtly remember that 10.x.y address space is widely used by CGNATs.
CGNATs should be using 100.64/10 instead of 10/8 to avoid this problem, but I don't doubt that there are significant deployments on 10/8 anyway.
The IETF really dragged their heels on CGNAT because they thought that IPv6 is easy™ (of course not, it's intentionally designed not to be "almost the same but wider" but include unworkable stuff like Mobile IPv6[1] which is just a fancy VPN) until they were forced to allocate 100.64.0.0/10 because some ISPs are not just using 10.0.0.0/8 but also US-DoD addresses (especially 11.0.0.0/8, because it's basically 10.0.0.0/7) as "private" addresses.
[1] Not IPv6 on mobile devices but a fully-owned IPv6 range that is supposed to be the address for a device regardless of where it is, see RFC 3775
I wanted to use 11.0.0.0 and call the company "Eleven," but by that time the DOD had given up the block for general use... GCNAT is perfect.
Are those usually visible to clients sitting behind routers though? I'm not super familiar but the things I'm seeing make it seem like that should only be visible IPs on the internal network of carriers which is not a place I am ever connecting from.
This is basically what I use tailscale & their magicdns feature for. I manage a few locally hosted jellyfin servers for myself and some family members, and its the same problem. I just added tailscale to them all and now I can basically do ssh parents.jellyfin.ts.net or inlaws.jellyfin.ts.net
I need to implement this type of thing for supporting networks of family members, but without the media server aspect - just computer/networking support. I'm looking for a cheap and reliable device that I can put in each home, to give the Tailscale "foothold". Do you happen to know of any tiny devices? I was thinking there must be something even cheaper than a Raspberry Pi to perform this single function at each location.
If they have an Apple TV, you can just install the app and use it as an exit node. I would check out the devices that are on their network currently, chances are you can use one of those.
An old micro pc from dell/hp/lenovo. They are often cheaper and more capable than Raspberry Pis. You can just put up a random Linux distro and it will work.
Use a pi zero it’s like $5
I was about to say that. This is what I do too.
The only drawback are routes - they won't work on the same CIDR (I mean the fact that you can say in Tailscale "if you want to reach the 192.168.16.13 device that does not support Tailscale, go through this Tailscale gateway"). For this I had to shift my parents' network to be able to access stuff like the printer, in a network that clashed with another one of mine.
The way we did it, roting is not a problem. Any Netrinos client (Windows, Mac, or Linux, including the free version) can act as a gateway. It assigns a unique overlay IP to devices on the local network that can't run software themselves, like cameras, NAS units, or printers, and handles the NAT translation.
Think of it like a router's DMZ feature, but inverted. Instead of exposing one device to the internet, each device gets a private address that's only reachable inside your mesh network.
This overlay approach is fantastic, but I do not think it exists in Tailscale.
How do you handle embedded devices that cannot install software?
In your experience, how often does Tailscale have to resort to an external relay server to traverse? I’ve had that out the kibosh on bandwidth/latency sensitive applications before.
> The gateway device performs 1:1 NAT. Traffic arriving for 100.97.14.3 is destination-translated to 192.168.1.100, and the source is masqueraded to the gateway's own LAN address.
Couldn't you tell the WG devices that 192.168.2.0/24 refers to the 192.168.1.0/24 network at customer A, such that 192.168.2.55 is routed to 192.168.1.55. Same for 192.168.3.0/24 referring to customer B.
I think this is what the article is getting at but I don't see the value in manually assigning an alias to each non-wg device, versus assigning an alias to the entire LAN.
It's not enough to set fake routes. You have to edit the addresses in the packets, so the end devices will receive them.
The problem there is you still need to keep track of the subnets. It works for a while, but it's quite complex. NAT is actually easier when you get into hundreds of sites.
Yeah so instead DNAT, use NETMAP on the gateway device to that LAN. (Sorry if I'm abusing the terminology, I only do this stuff like once a year for homelab.)
eg this is what I'm currently using to alias my home network
With other wg peers getting a 192.168.150.0/24 entry in the AllowedIPs for this gateway (if needed).The suggested solution involves using the CGNAT /10 in conjunction with a VPN, but I've actually seen someone do this, and still have problems with certain end users where their next hop for routing also involves a router with an IPv4 address in the same space, so it's not really bulletproof either. We may as well consider doing other naughty things like co-opting DoD non-routable /8s or the test net in the RFCs you're not supposed to use, because basically anything you pick is going to have problems.
That does not happen here. The CGNAT addresses are in the VPN tunnel. And the tunnel connects private devices end-to-end. The LAN packets never see the Internet. They are inside the WireGuard packets.
This is what the NETMAP target in iptables is for - map an entire subnet to another subnet, including the reverse. We were doing this 20 years ago for clients trying to on-board other companies that they'd bought. It's horrible, but it does solve the problem in a pinch.
I feel like this is really only an issue with true site to site VPNs. Client to site shouldn't have this issue because the VPN concentrator is like a virtual NAT.
The best strategy might be to maintain the ability to easily reassign the network for a site. If every site is non-overlapping the problem does become trivial. I'd much rather fight a one time "reboot your machines tonight" battle than the ongoing misery of mapping things that do not want to be.
One step beyond this is the multi-subnetted network on each side. You get the DNAT working, but then suddenly the app gets more complex over time and suddenly you're calling 192.168.2.x, which leads to async routes. Some traffic works, some traffic works one way, and other traffic disappears.
Then you as the client/app manager pull your hair out as the network team tells you everything is working fine.
At the risk of ruining my solution I moved my lan sub net into the
172.16.0.0/12 block
This is used on virtual private clouds and is not publicly addressable.
since switching to this I have not had any collisions.
Shameless plug - this is exactly the same problem that our team had when we had to maintain a bunch of our customer's servers. All of the subnets were same, and we had to jump through hoops just to access those servers - vpns, port forwarding, dynamic dns with vnc - we've tried it all. That is why we developed https://sshreach.me/ - now it's a click of a button.
The initial idea started as a bunch of ssh tunnels. Been doing that for years. But WireGuard seemed a better solution at scale, and more efficient. When I first saw WiteGuard, it blew my mind how elegantly simple it was. I always hated VPNs. Now I seem to have made them my life...
Your website landing page is great. No stock photo hipsters drinking coffee, no corporate fluff amid whitespace wasteland. Just straight to the point. Rare sight today.
Like Newt (https://github.com/fosrl/newt)?
I decided to learn IPv6 recently and I'm pleasantly surprised how simple and elegant it is. Truly a joy. Highly recommend, if you've never worked with IPv6 to try it. It's like discovering a bidet.
> But the moment two sites share the same address range, you have an ambiguity that IP routing cannot resolve.
Writing PF or nft rules to NAT these hyper-legacy subnets on the local side of the layer3 tunnel is actually super trivial, like 20 seconds of effort to reason about and write in a config manifest.
Like written the article, a device on the customer site is required. At that point you might as well deploy a router that has a supportable software stack and where possible sober IP instead of legacy IP.
.
I have been running IPv6-only networks since 2005 and have been deploying IPv6-only networks since 2009. When I encountered a small implementation gap in my favorite BSD, I wrote and submitted a patch.
Anyone who complained about their favorite open source OS having an IPv6 implementation gap or was using proprietary software (and then also dumb enough to complain about it), should be ashamed of themselves for doing so on any forum with "hacker" in the name. But we all know they aren't ashamed of themselves because the competency crisis is very real and the coddle culture let's such disease fester.
There is no excuse to not deploy at minimum a dual-stack network if not an IPv6-only network. If you deploy an IPv4-only network you are incompetent, you are shitting up the internet for everyone else, and it would be better for all of humanity if you kept any and all enthusiasm you have for computers entirely to yourself (not a single utterance).
I won't name the 2 large telecoms I know, that don't support IPv6 being used by customers - if you get L2VPN, L3VPN, other typical services etc. it will be IPv4-only. Of course you can buy a wave and do whatever you want with it :-)
Support for IPv6 is notoriously bad in residential modems. They can barely run IPv4. In an enterprise, you can do it properly. But here we are stuck with the junk the ISP gave out. Customers don't care. You have to work with what you've got.
>Support for IPv6 is notoriously bad in residential modems.
No? Over here at (South) East Asia we have been deploying IPv6 for nearly a decade now. The users are getting their IPv6 connectivity. Before someone jumps out and shouts SeCuRiTy: the firewall is enabled by default.
I am not saying the support is perfect. I know some people moan about lackluster IPv6 configuration in many routers. But for 90% of residential internet users (who care about pretty much nothing but the ability to watch YouTube and browsing social media), it damn sure is.
[dead]
I don't have enough time for that.
[flagged]
Can we please just use ipv6? PLEASE
no because it wastes 24 bytes per packet!!!11111