PMTU discovery on the Internet is generally unreliable. Very few people understand that it exists, and even fewer understand how it actually works. Most ADSL (PPPoE) providers rewrite the TCP MSS on TCP SYN packets traveling over their network to account for the PMTU discovery brokenness. [1] You see the same thing happen with VPN connections where the PMTU is effectively reduced by the size of the overhead for the encapsulation protocol.
I agree, and when I worked in that world it was a huge pain in the ass. Users don't care about RFCs though, they just want the Internet to work, so you end up doing something that's pragmatic but kludgey. As I pointed out in another comment, Google is responding with a TCP MSS of 1430 (so assuming a PMTU of 1470). It's just what you do so you can get on with your business.
"A packet-filtering router acting as a firewall which permits outgoing IP packets with the Don't Fragment (DF) bit set MUST NOT block incoming ICMP Destination Unreachable / Fragmentation Needed errors sent in response to the outbound packets from reaching hosts inside the firewall, as this would break the standards-compliant usage of Path MTU discovery by hosts generating legitimate traffic. "
That would be great, next tell the folks at SBCGlobal to fix their damn network as well. I don't know how many folks we've had to 'patch' by manually walking the MTU down on the local router until packets actually get through. It really really sucks and leads to sending way more small packets than needed.
I'm not sure I completely understand your situation, but turning down the MTU on an intermediate router is rarely a good fix for PMTU issues. If you have web servers, you may consider turning down the MTU on the Internet facing interfaces. Even if you turned it down to say 1400, you'd only have around 5% more packets ... probably not enough load increase to melt your routers.
In fact, I just ran some test TCP connections against Google, and their web servers respond a TCP MSS of only 1430, suggesting they are assuming a PMTU of 1470. That sounds pretty pragmatic to me and I wouldn't be surprised if other large Internet companies did a similar thing.
The way this expresses itself is some web sites work and some just hang. Looking at the traffic with wireshark you see packets go in but they don't come back. But other web sites work just fine. So you start ratcheting down the MTU size on the outbound router until the non-functioning web site starts returning your calls. Doing ping's with various sized packets can (if the server is responding to pings) also identify the longest packet you can send before someone between you and them decided they want a different fragment size.
There should be some sort of open-source testing environment for common breakages. For example, maybe a bunch of VirtualBox VM's running Linux. Maybe a unit test for the problem mentioned in the article sets up machines A, B, FW, and C connected A <-> B <-> FW <-> C, has B fragment packets, and has A perform path MTU discovery to C. The firewall configuration under test goes on FW and running the test will catch this problem.
With a good enough framework of this type, all the testing could be done "out of the box" so all you have to do is set up a disk image or IP address of a firewall box to test, and the testing is fully automatic. The firewall under test can run any OS or firewall that will run as a VirtualBox client -- or even be its own box connected to the testing machine's Ethernet port. Heck, if you had machines on both sides of some third-party you don't control, like your ISP, you could even use it to probe their network configuration for issues without any special cooperation from them.
If the test suite gets good enough, maybe eventually pressure will build on vendors to make their products pass and we'll see firewall brokenness start to disappear.
As well as cloud services like AWS, such tests could be used by Linux distros, operating system vendors, and network equipment manufacturers.
I'd build it myself, but I'm not a networking expert and I'm not particularly enthusiastic about becoming one.
Much like Google did with SPDY, I suspect they can (and at some point some big player will, be it google or someone else) release a TCP replacement that, unlike IPv6, is actually useful if you only have an IPv4 link between endpoints; and it will be this protocol that will eventually replace IPv4 rather than IPv6.
First, don't read apenwarr for the info, read the paper.[1] Unfortunately, that still won't carry you that far because there's a whole lot of "community knowledge" that is carried in DS papers that you don't get unless you read a lot of them or are in the community.
CCN is not an actual proposal. It's a pie-in-the-sky networking redesign.
Do you know what would be required to implement CCN? Go read the paper. The hardware cost alone is insane. This is not coming any time soon. Be prepared for IPv6 for the next 20 years at least.
I'll take your word for how pie-in-the-sky CCN really is, I skimmed through the paper without a critical eye. However ...
> Be prepared for IPv6 for the next 20 years at least.
I have, for the last 10 (or more?) I think it was 1999 when I was thinking "I should probably implement IPv6 in this project, because next year it's going to be everywhere".
I think you mean "be prepared for IPv6 to be the up-and-coming protocol for the next 20 years at least".
Maybe not CCN, but I suspect it's more likely that a killer-app with it's own tunnelled-in-IPv4 protocol is more likely to take over than IPv6. BitTorrent had a chance to do that, but didn't. I can't point to the next one myself.
Because IPv6 breaks hard if ICMP is filtered. It depends on a bunch of newly allocated message types and replaces things like ARP and (some functions of) DHCP with broadcast ICMP.
There is tons of little crap like this in the low level details of IPv6. "We don't like that people do X, so we will force them to stop"
IPv6 no longer supports fragmenting on routers. That means if you don't want to be stuck with the default minimum MTU of 1280 (which you really don't want to for low-latency applications) you need to support Path MTU Discovery, which in turn requires ICMP to go unhindered across a large number of different networks between you and the receiver.
OK so I can see how it violates standards. How many of the millions of users that send traffic through EC2 does this affect however? I can see how they would be reluctant to mess with Firewall rulesets. Even if it they only apply it to new users that would mean fragmentation ... Keep it simple stupid. Again it depends on how many users this affects and from the sounds of the blog post - vanishingly few
In my experience this is the rule rather than the exception, most firewall configs are broken in some way and there are often several firewalls on the path. I turn them off where circumstances allow.
If we just accept "networks are unreliable and sometimes broken" as a fact of life, things will never get better. I applaud the unsung heroes who are finding and fixing the actual root causes of lower layers of our networks. Other important networking issues that come to mind are bufferbloat and IPv6 brokenness.
I'm sure it will be fixed if this stays on the front page for a while, so be sure to upvote the article.
>If we just accept "networks are unreliable and sometimes broken" as a fact of life, things will never get better. I applaud the unsung heroes who are finding and fixing the actual root causes of lower layers of our networks.
As developers, this should always be a fact of life. True, we should strive for making the networks perfect, but at the end of the day, these things still need to be accounted for.
Because you don't have control over a client deciding that 50 cent network cards are "good enough" for their deployment even though they've demanded five 9's uptime from your software.
Because you can't know when someone's going to spill beer on the switch.
Because someone's just going to pull the wrong cable.
Because the water company accidentally cut the line into the building.
You're mostly talking about reliability at the physical layer, which as you say is never going to be perfect. Csense is talking about reliability at the link layer and above, which is infinitely more attainable.
If you can't guarantee that every layer below you is absolutely reliable, then you need to assume that everything below you might be broken, and that you need to handle it. You can't start with the mindset that everything works below you, and have everything above you also work fine.
The fact that we have people with the mindset that everything below them is broken is the entire reason that these kinds of issues get detected and fixed.
Yeah if you try and work around broken PMTUD though you're going to mess up your application layer protocol SO bad...
EDIT: I suppose you could rewrite TCP MSS on your own firewall or drop MTU on all web servers.... but of course if you're going to reconfigure your firewall / intefaces, you may as well just fix the problem - which was caused by poor device configuration in the first place.
You also can't know when someone between you and the other endpoint has configured something stupidly. :)
As a sysadmin, I do my best to make the network as damn near perfect as possible; but as a developer, I still generally assume the network is unreliable and plan accordingly.
I think it SHOULD be accepted as a fact of life. That way we can then learn to improve our designs of applications and processes to be more tolerant to network faults and unreliability, which is a good thing.
Basically, Hurricane Electric and Cogent (two major transit providers) are refusing to exchange IPv6 traffic. That breaks stuff. For me the problem manifested as a user facing a 20 second delay whenever he ssh'd in (over IPv4). (I'm on Hurricane Electric. The reverse DNS for his IPv4 address is served by Cogent DNS servers. Reverse DNS lookups were trying to access the Cogent name server over IPv6 and had to time out before IPv4 was tried.)
Other than that though, I actually haven't encountered very much IPv6 breakage.
No, mainly because it would be business suicide for whoever is involved. Especially if neither party is willing to pay for transit to the other... IPv6 is still small enough that it doesn't yet force one providers hand over the other.
That being said, if you are single homed on HE only then I would suggest you purchase transit elsewhere as well, or find a datacenter that has multiple transit providers. Being single home is dangerous.
The real question is going to be how long can Cogent hold-out. HE is willing to provide them free IPv6 peering, why not accept it so that their customers can get a full IPv6 table...
People often bungle their firewall rules because they don't know any better, but Amazon continuing to willfully fuck up TCP for all of AWS is a pretty large issue for the functioning of the net at large.
It absolutely does stop SSH or Web traffic, if the network path goes through a link with MTU < 1500 and the connection comes in with MSS > PMTU - 40. But only, as the post says, once you start sending a lot of data in one TCP segment.
I've had more than a few cases where I couldn't SSH into a system at a hospital or clinic because the VPN/firewall/whatever that their connection went over rejected packets with too high of an MTU. Generally you'll get your SSH connection and it'll hang in the middle of the MOTD.
Because people like AGL, CPercival, and me do care, i.e., we develop network applications, care about users, and prefer to avoid rather than troubleshoot random brokenness.
Publicly expound on how you don't care about those users and they are SOL because they are in the minority. It won't matter because it doesn't affect the rest of your userbase, right?
When working on a packet analysis system for a VoIP company, I found that around 1% of incoming UDP traffic was fragmented, the size being around 576 bytes.
I don't know how representative that was (10s of thousands of users, but large users generating more traffic than smaller ones), but < 0.01% is probably on the low side.
1. http://www.cisco.com/en/US/docs/ios/12_2t/12_2t4/feature/gui...