Intuitively, I would say that this is actually undefined behavior (it would probably be difficult to expose a wrong behavior in practice though).
In C specs, I found 6.5.2.2, paragraph 9:
> If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.
It's worse than that. This guy takes a void* function and casts it to a char* function, then passes it a char**.
void (*name)(char *ptr);
typedef void (*name_func)(char *ptr);
void target(void *ptr)
{
printf("Input %p\n", ptr);
}
char *data = "string";
name = (name_func)target; // Illegal: casting fn that takes void* to a fn that takes char*
name(&data); // Illegal: passing a char** into a function that takes char*
Before someone mentions qsort(): the comparator function really is supposed to take a void*, and inside the function, you re-cast the void* argument to a pointer type of your desire. If you don't do it in that order, you're using it wrong.
The C standard mandates that a “pointer to void shall have the same representation and alignment requirements as a pointer to a character type”. In consequence, the same holds for an array of void pointers vs. an array of char pointers. The code therefore seems valid to me, and furthermore at no point is a function called with an argument type different from its formal parameters.
In the GP example, a char** is passed where a char* is expected. That is clearly invalid.
I was more referring to how, in the K&R example, a function of type "int (*)(char *, char *)" is cast to "int (*)(void *, void *)", in contradiction to what they said here:
> Before someone mentions qsort(): the comparator function really is supposed to take a void*, and inside the function, you re-cast the void* argument to a pointer type of your desire. If you don't do it in that order, you're using it wrong.
In contrast, the K&R example is an example of explicitly undefined behavior in the C standard:
The behavior is undefined in the following circumstances:
- A pointer is used to call a function whose type is not compatible with the pointed-to type (6.3.2.3).
The second edition was released in 1988, and it was based on a draft of the first ANSI C standard, and even then, the line stating that this was undefined behavior was already present.
> A pointer to a function is converted to point to a function of a different type and used to call a function of a type not compatible with the original type (3.3.4).
Ah, I missed that. However, footnote 41 states “The same representation and alignment requirements are meant to imply interchangeability as
arguments to functions […]”, which could be taken as implying compatibility of function types, though the normative wording doesn’t directly support that.
In C, all pointers are compatible to a void pointer without casting though (e.g. assigning a char pointer to a void pointer or the other way around is completely legal - I think that was the main reason why void pointers were added in the first place - it's basically an 'any pointer'). It's only C++ where this is an error (which is weird on its own, why even allow void pointers in C++ when the only reason they (presumably) exist doesn't work anymore).
The above code still doesn't compile in C because C complains about the function signatures being incompatible, even though the only difference is the argument type which itself is 'equivalent' in C - which I guess could be interepreted as an edge case in the C spec? (IMHO the function prototypes should actually be compatible).
"Compatibility" has a specific meaning in C which isn't the same as what you're describing here. What you're talking about is implicit conversions without narrowing.
The C standard is quite loose about pointer requirements. Conversions between data pointers or between function pointers are allowed, but conversions between each other are not guaranteed. You can read the nitty gritty here: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
POSIX defines stricter requirements for pointers than the C standard. All pointers are compatible, so they have the same size and representation, and conversions between function pointers and void* are explicitly allowed.
you can convert a short to an int without casting and it will roundtrip without loss of data; similarly you can roundtrip any data pointer through a void pointer without loss of data. It doesn't mean that an int has the same representation of a short or a void* has (necessarily) the same representation as any other pointer.
In C++ any pointer is also implicitly convertible to a void *, it is the reverse implicit conversion that is prohibited as it is not safe in the general case.
For consistency C++ should also prohibit implicit narrowing conversions (e.g. int to short ); I guess this was thought to break too much existing code and it is generally safer than a pointer conversion (although potentially lossy the behavior is fully defined). Many compilers will warn about it, and narrowing conversions are now prohibited in braced initialization.
Yes. Simply put, void pointers and struct pointers (the actual breaking example in Curl) are not guaranteed to have the same byte-level representation, by the C standard. (For example, a struct pointer may always be aligned and therefore use fewer bits than a void pointer.) Passing one to a function that expects the other may result in arbitrary breakage.
This is one of the very rare cases in C where something is technically Undefined Behaviour, but in practice works and is recommended.
The typedef struct trick, is very common idiom that creates _more_ safety, not less. All reasonable compilers should (and do) support this. It is sad that the ISO standard is not in line with reality at all times. (I say that as a member of the wg14 and the UB study group)
I Recommend Daniel keep his typedef struct definition, and then have an ifdef to revert to the void definition for when Clang does its UB sanitizer. While checking for prototype discrepancies is a very good thing to automate, Clang should add an exception for this.
There is no problem with using the typedef struct trick. The problem is assuming that such pointers have the same underlying representation as void pointers. Structs typically having stricter alignment requirements (whereas void* must be able to point to any byte) is one reason why their representation may differ from void*. This is why only [[un]signed] char* is guaranteed to have the same representation as void*.
Alignment only matters when you use a pointer. This is a question of if technically non compatible signatures are UB by them selves. Unaligned access is UB for sure, but even if everything is perfectly aligned they may still be UB according to the standard.
Example:
Lets say you have a function that takes a pointer to a 32 bit integer.
void my_function(int x)
But in an other file you declare it as a function taking a 32 bit float, and call it:
This is very likely to work in almost all implementations. float and int have the same size and same alignment requirements, but the standard may still say its UB. So it is technically UB, but may be relied upon in practice.
> I say that as a member of the wg14 and the UB study group
Since you have experience in this area, do you know how likely it is that something like this could be resolved? I.e., if someone proposed "just make this defined", how likely would the C standards body be to agree and do so?
These things can be fixed and often are. Some one just needs to write a paper proposing a change. I might in fact do so on this issue. It should be resolvable.
Lots of code has been broken because people have written UB code and then optimzers become smarter and vreak the code.
This category of we call ”usless UB” are things that wont help optimizeayions, and all major compiler have to support beacause if they dont, their users complain too much, can be relied on.
Unfortunately its very hard for the average use to know what UB can be relied on (there isnt much).
Awesome writeup. Always interesting to read what Daniel has to say.
I think the fact that it turned out that he was wrong (and UBsan was right, as usual) is a great testament to the shortcomings of C.
Lots of people - both inexperienced and very experienced - celebrate it for being "simple" and "close to the hardware", but the truth of the matter is that it is precisely not close enough to the hardware for people who _know_ what the hardware is doing to be able to do what they expect, and it's too close to the hardware to be able to be able to ignore it.
Lots of experienced C programmers (and - guilt by association - C++ programmers as well) run into UB because they have clear expectations of the compiler. I.e., they know what the compiler should generate, more or less, and C is just a convenient notation. But compilers don't live up to those expectations, because they don't actually compile your code for the hardware. They compile it to the virtual machine abstraction defined by the standard, which very often works differently from any real architecture, and then translate that into machine code. Even though there is basically a single set of semantics that every single "relevant" (mainstream) architecture implements. This is a holdover from when C had to target architectures that are 100% irrelevant today.
Everybody's favorite example is signed integer overflow. In both x86-64 and ARM64, that just works - two's complement is the only relevant implementation, so there's no issue. But `int` in C and C++ is not that.
Almost every single common UB pitfall has reasonable behavior at the assembler level for every mainstream architecture, and almost every single niche architecture.
C gives you the illusion of being close to the hardware, but in actual reality the hardware is several steps removed, so if you want to leverage your knowledge of the hardware, calling conventions, assembly, or other low-level details, you have to go out of your way to work around the C standard.
(Aside: We need new languages to tackle this, and I coincidentally happen to like Rust. Lots of people coming from C or C++ are irritated and frustrated by Rust, but 99% of the time it's because Rust gives you a compile error where C would give you UB. This is one example of that out of thousands.)
Many of such issue were also fixed in languages like Ada and Modula-2, among others, which C and C++ folks used to call "programming with a straighjacket".
Many folks will complain about anything that breaks their beloved illusion that C is a fancier macro assembler, except even proper macro assemblers have less UB than regular C.
Being close to the hardware isn't a dogma. The point of being close to the hardware is to be able to write code that is (nearly) as perfomant as it could be if you used assembler. Undefined behavior that let's the compiler better reason about your code like signed integer overflow is entirely compatible with that goal. If anything, unsigned overflow should also be UB unless you explicitly tell the compiler that an operation should wrap, which really is not something you want in the common case where any overflow already means you have a bug.
Wouldn't a faster and easier solution for existing code be to come up with a standard set of behaviours based on common architectures and make that a flag that can be switched on so expectations can be met (and call it "expected-C" or for the pun "ocean")?
AFAIK the reason for signed overflow still being UB isn't any exotic hardware property but because it enables some specific optimizations. Whether those optimizations are worth the trouble is up for discussion I guess.
No, it used to be the case that there were architectures where signed integers were represented as 1s complement, so portable code could not rely on signed integer overflow wrapping around (there would either be 2 bit patterns representing zero, or sometimes the all-ones pattern had special meaning, like a trap).
Using this type of UB is a "relatively" new thing (GCC started doing it in the 00s, which broke a lot of stuff in the Linux world, IIRC).
It _is_ true that somebody did the research (can't find the source right now) and found that defining signed integer overflow as wrapping did indeed make some code run slower. I'm skeptical that it matters.
That's why I wrote "still". AFAIK both C++ and C now expect integers to be in 2s complement and made unsigned overflow 'defined', but at the same time kept signed integer overflow as undefined behaviour.
C does give a lot of low-level control, but it is still an abstraction - as it should be.
Signed integer overflow is something were I personally like that it is UB, because I can turn on UBSan and find overflow bugs. If it were defined to wrap-around I could not do this.
Does anyone know of languages that achieve this? I'm interested, I'm currently implementing a project in x86 assembly for this reason, and am happy to try a higher level language.
Depending on what you mean exactly I think Rust can be considered pretty close to the hardware. I.e. there's usually an obvious straightforward translation of each line of source code to an implementation in assembly language (which may not be what's actually emitted in practice due to optimizations, but the same is true in C).
There are of course some higher-level features like trait-based generics, so it's not really as close to the machine as C, but it's a lot closer to C IMO than something like Java (or even C++).
Ada, Modula-2 (both available on GCC), D (available on its reference implementation, GCC and LLVM), Swift, Object Pascal (Free Pascal, Delphi, Oxygen), Oberon variants (Oberon, Oberon-2, Oberon-07, Active Oberon, Component Pascal), Zig, Nim, Odin,...
C++ is further than rust in my opinion. Vtables that support inheritance, as well as stack unwinding for exceptions, are pretty complicated and totally implicit in C++. Okay rust also technically has unwinding for panics to be fair but it’s rather unusual in practice for programmers to use panics to mean anything other than “crash the program now”.
That’s fair, trait objects do cause a table to be generated, but they don’t support inheritance and subjectively I think they’re used less often than the OOP features of c++ that lead to vtable-based dynamic dispatch. (Traits are extremely common in rust, but dyn trait objects somewhat less so)
For signed overflow it's fascinating, Herb Sutter (WG21 convenor, Microsoft employee) writes that checking would "incur unacceptable costs".
Now, C++ programmers, were you consulted about these costs? Herb insists they're "unacceptable" but he provides no further information as to what the cost actually was, or who decided whether that cost was acceptable, much less how this could generalize across a wide variety of domains and platforms.
What's Herb's answer? You might hope that Herb would say OK, we'll provide wrapping for these types by default, it's not checked arithmetic but at least it's not UB. Nope.
C & C++ lack a particular sort of behavior that could solve a bunch of problems.
They've got undefined behavior: The standard imposes no requirements. Anything can happen. Assume the worst.
They've got unspecified behavior: Only covers behavior caused by an unspecified value or where the standard provides multiple choices, where the implementation need not document which choices are made in which situations.
They've got implementation-defined behavior: Unspecified behavior that the implementation must document.
They don't have a category for "Undefined behavior but the implementation must document". A lot of what is currently undefined behavior could better be put into this category, if it existed.
C++ 26 will (almost certainly, it's in the draft) add Erroneous Behaviour.
EB is well defined but definitely wrong, so it's a way for the standard to say:
Do not allow this to happen, but if you do, the consequence is definitely that.
The specific EB in the C++ 26 draft is the value of default uninitialized primitives. So e.g. int k; std::cout << k << "\n";
In C++ 23 and previous versions that's Undefined Behaviour, maybe it prints the lyrics to the National Anthem of the country where the compiler ran? Maybe it deletes all your files. But in C++ 26 the Erroneous Behaviour is that there's some integer value k, which your compiler vendor knew (and might tell you or even let you change it) and it prints that value, but you're naughty because this is definitively an error when it happens.
With GCC/clang can just add checking with -fsanitize=signed-integer-overflow -fsanitize-undefined-trap-on-error.
For my main software project, which is some numerical software for magnetic resonance imaging, this adds 12212 checks and the optimizer reduces them down to 3803. But I haven't done benchmarking yet, but I would guess that for most software it would not matter.
Basically the behavior is hardware-dependent, and nobody wants to mandate that C++ compilers generate a ton of extra instructions on hardware which does not behave a particular way.
Of course you can define your own checked integer types, using inline assembly to check the overflow flag where available.
Just so we're clear, yes, it's "hardware-dependent", but literally every single architecture and CPU model does the same reasonable thing, which is to wrap into the negative.
Any architecture that doesn't use 2s complement is so esoteric by now that it does not make any sense for a general-purpose C compiler to pretend they exist.
I think this is more of an edge case than you give it credit for.
(External linkage) functions, and their callsites, are quite special as they straddle the boundary between actual machine and virtual machine.
The calling convention makes fairly strict requirements of what that must compile down to, since caller and callee could be dynamically linked objects compiled by different compilers or even entirely different languages.
If you define a function that takes a void* parameter it will read a pointer from the RDI(?). If you pass a char* it will pass the pointer in that same register.
Now it must be said that if the compiler can see both the definition and call, then it's free to do whatever it wants e.g. inline or something.
Sure, I mean, my objection is with the pretense that a reasonable calling convention in 2024 could decide to represent void* differently from any other T. It makes total sense that C programmers expect pointers to be (transitively) convertible to/from void, so the fact that they aren't convertible in this way without a trampoline means that the standard contains surprises for even very experienced developers.
I postulate that almost all UB in the wild comes from the Standard diverging from (often very reasonable) expectations, and I see that as a big problem with the standard, at least as long as compilers can't reliably detect the problem at compile-time. (And yes, C++ is even more problematic here.)
I think one of the reasons Zig exists is that it contains far fewer surprises. The reason Rust exists is that it does a much better job at preventing and containing such surprises.
Every C compiler I ever used will tell you that void()(void) is not convertible to void ()(char). If people still do it then they are a bit on their own. But how is this then different to Rust's "unsafe"? (of course, there is other UB compiler do not tell you about, that this seems a bad example)
If BUILDING_LIBCURL isn’t defined that tells code “CURL is identical to a struct named Curl_easy”. If it is defined, that also tells code what fields it has.
Interesting problem. The typical solution in C++ to deal with type erasing function pointer types is to go through a trampoline function:
struct X {};
void use_x(X*);
using F = void(void*);
void bar(F* fn, void* y) { fn(y); }
template<class T, auto fn>
void trampoline(void*arg) { return fn(reinterpret_cast<T*>(arg)); }
X x;
bar(&trampoline<X, use_x>, &x);
In plain C there is no way to generate the trampoline at the point of use in the same way template instantiation works, but it can be generated by a macro at global scope.
Yes, it's very much undefined behavior. As I recall, GTK's glib does this all over the place for signals.
edit: I'm not advocating that this being UB is fine. I don't expect compilers to exploit this for optimization, because so many projects rely on this working. There might be room to extend compatible function types to make this defined.
> In 2016 I wanted to change the type universally to just typedef struct Curl_easy CURL; … as I thought we could do that without breaking neither API nor ABI.
This seems to be the obvious solution and how most libraries define their opaque handles. It doesn't break a guaranteed API and it doesn't break the ABI any more than than only using it when building the library - and you can check that it doesn't break the ABI on any platform where you want to guarantee ABI stability.
In the real world, we often want to avoid breaking people's code even when people rely on something that's not guaranteed by the API docs. It seems like Daniel's goal isn't to only be API-compatible in a technical sense (namely that perfectly written code which carefully avoids using anything in a way that's not explicitly guaranteed to work), but rather to avoid breaking people's existing code. I can respect that.
Yes but that is always a balancing act if you want to make any change at all since users can theoretically depend on any possible implementation detail or even on outright bugs.
Correct. It's always a balancing act. That means hard rules like "we will do any change which doesn't technically break any promises explicitly made in the documentation in a patch release" aren't appropriate, it's always a value judgement (and sometimes you get it wrong and revert the change once you realize that people were affected more than expected, as happened in the case of curl).
Ok, this is UB, calling a function pointer through a different type than its definition is a pretty clear example of UB. The problem here is that there's a confusion between "void * and char * are implicitly convertible in C" and "void * and char * are the same". The latter is true for many platforms (especially older ones) but not all (I think there were platforms where functionally they had `typedef char void`). There's a side note of conflating "this has defined behavior on my platform that is stable and works for me" and "it's not UB if it's stable and works on a platform", just like integer overflow is UB despite being entirely defined behavior on every platform under the sun.
Anyway, if folk are curious there are many platforms where not only can the representation of `void()(void)` and `void()(char)` be different - even if pointing to the same function - but the representation of even just the data pointers void* and char* may not be the same, again while pointing to the same memory.
For example, on platforms with pointer authentication function pointers are generally (I would say "always" but in principle it can be avoided) signed, and in some configuration the type of the function is incorporated into the function. Calling the function pointer requires authenticating the pointer, and authenticating the pointer requires that the call site agrees on the type of the pointer because otherwise the signature fails.
Absent actual pointer auth hardware you could imagine someone implementing this as some kind of monstrosity like this (very hypothetical, unpleasant, and footgun heavy) horror:
and you can immediately see that if you had code that used these but disagreed on the type of the function you'd have a bad time. With compiler+hardware pointer auth this is just handled transparently.
In principle a pointer auth environment could apply this type discrimination logic to data pointers as well, but I'm unaware of any that do so implicitly. But if a platform did do so, then the incorrect type of the parameter would mean you would fail inside the function, if you were able to call it (say if you weren't using a function pointer, but had mistyped the prototype).
Similarly, in other environments, the pointer may be directly aware the type being referenced, and I believe that CHERI supports this, in which case even if you could call the function pointer when attempting to read the pointer I believe it would fail.
Having got here, you might be saying "but hang on C says void* and char* are the same", and we go all the way back to my first sentence where I said "are implicitly convertible" :D
On plenty of systems casting from one pointer type to another is an entirely source level feature and once lowered is completely invisible. But in the environments we're discussing
(Type1*)pointerToType2
Under pointer auth it requires re-signing the pointer (you have to auth the original value to verify it, and then sign the result according to the new schema), and under CHERI I believe there are instructions for controlling how a pointer is tagged.
But the important thing is the C does not say they are the same thing, just that the conversion is automatic, just like numbers and bools, or numbers and bools in JS, or numbers and strings in JS, or objects and strings in JS, or nothing and strings in JS, or .... :D
Sorry I’m not clear what you’re saying - because I don’t know exactly which sentence(s) in my giant wall of text you’re referring to :D
I did try to simplify things as well rather than focusing on specific language technicalities, because just going into arbitrary amounts of technical detail didn’t seem like a good way to explain things, and I’ve found can result in people thinking you’re focusing on those “technicalities” rather than “real” issues, without understanding the technicalities are the root of the “real” issues.
This is not a rant, but a well laid-out description of something a widely used software ran into as a result of a change in the compiler they use.
The #ifdef they had in place was a bit hackish, but I wouldn't call it abuse. It is a typical construction for C to do stuff like that which in general isn't without risk, but they have used it without any problems for years. The whole point of the blog post is to start a discussion whether sth. like this should rightfully be flagged as "unexpected behavior" or not.
It's not well laid out. The examples are malformed/illegal and the ifdef thing is stupid.
The author admits to not being a C undefined behavior expert and yet acts like they might know better than a tool made by such experts.
Looking up the rules and verifying the shown snippets takes at most 30 minutes at a leisurely pace, the author could have saved themselves the embarrassment.
I'm not going to write a blog post about how I didn't expect a color spectrometer pointed at the sky to say "BLUE" because I thought it might have been purple, "although I'm not an expert in wave lengths".
> The author admits to not being a C undefined behavior expert
At this stage, I would seriously doubt the credentials of anyone who claims to be a C undefined behaviour expert. Saying "I'm not a C UB expert" is just a realistic acknowledgement that UB is hard and we will get it wrong at some point without realising. The approach of having an automated tool tell you when UB is present is very sensible.
The ifdef thing certainly isn't "stpuid". It's not good design, you wouldn't design it that way if you made libcurl from scratch today, but it makes sense as a solution to the problem of, "we can't change the type of CURL* in the public API, but internally, it ought to be defined as a pointer to a struct". If it was well-defined behavior, it would probably have been the best solution possible given the constraints.
Casting function pointers like this can break Emscripten [1]. In libxml2, I fixed all these issues 7 years ago. It can be painful, but "it worked for 28 years" is not an excuse.
I believe AIX C++ name mangling includes function argument type information (with CV qualifiers!) so this is a real-world case where this does actually break. I suspect curl does not compile with the C++ compiler though.
It's literally the opposite of "works on my machine" because it's in curl (which is most likely the single most widely deployed code base in the world).
That is the thing, people keep treating it as a portable macro assembler, when it stopped being so decades ago, and those folks haven't yet got the memo.
> This construct works perfectly fine in C
Intuitively, I would say that this is actually undefined behavior (it would probably be difficult to expose a wrong behavior in practice though).
In C specs, I found 6.5.2.2, paragraph 9:
> If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.
We might discuss whether
is "compatible" with but I think it isn't, since: fails to compile with the error message: The compiler explicitly says "incompatible pointer type".Same for:
> it would probably be difficult to expose a wrong behavior in practice though
Emscripten breaks if you cast function pointers: https://emscripten.org/docs/porting/guidelines/function_poin...
Good point, interesting that I never ran into this issue in real world code bases so far (and I build a lot of code with Emscripten) :)
It's worse than that. This guy takes a void* function and casts it to a char* function, then passes it a char**.
Before someone mentions qsort(): the comparator function really is supposed to take a void*, and inside the function, you re-cast the void* argument to a pointer type of your desire. If you don't do it in that order, you're using it wrong.Ironically, in K&R, they did exactly this for casting comparator functions for their own version of qsort.
The C standard mandates that a “pointer to void shall have the same representation and alignment requirements as a pointer to a character type”. In consequence, the same holds for an array of void pointers vs. an array of char pointers. The code therefore seems valid to me, and furthermore at no point is a function called with an argument type different from its formal parameters.
In the GP example, a char** is passed where a char* is expected. That is clearly invalid.
I was more referring to how, in the K&R example, a function of type "int (*)(char *, char *)" is cast to "int (*)(void *, void *)", in contradiction to what they said here:
> Before someone mentions qsort(): the comparator function really is supposed to take a void*, and inside the function, you re-cast the void* argument to a pointer type of your desire. If you don't do it in that order, you're using it wrong.
In contrast, the K&R example is an example of explicitly undefined behavior in the C standard:
The behavior is undefined in the following circumstances:
- A pointer is used to call a function whose type is not compatible with the pointed-to type (6.3.2.3).
> the K&R example is an example of explicitly undefined behavior in the C standard
You can’t blame K&R C for that. It predates “the C standard” by over a decade (1978 vs 1989)
When it was written, what K&R said was C.
The second edition was released in 1988, and it was based on a draft of the first ANSI C standard, and even then, the line stating that this was undefined behavior was already present.
http://jfxpt.com/library/c89-draft.html#A.6.2
> A pointer to a function is converted to point to a function of a different type and used to call a function of a type not compatible with the original type (3.3.4).
The example is still the same in the second edition of K&R, which is based on C89.
Ah, I missed that. However, footnote 41 states “The same representation and alignment requirements are meant to imply interchangeability as arguments to functions […]”, which could be taken as implying compatibility of function types, though the normative wording doesn’t directly support that.
Good point. It would be nice if the C standard were slightly clearer here.
> name(&data); // Illegal: passing a char* into a function that takes char*
I assume this is just a typo in the article. He probably meant `name(data)`.
In C, all pointers are compatible to a void pointer without casting though (e.g. assigning a char pointer to a void pointer or the other way around is completely legal - I think that was the main reason why void pointers were added in the first place - it's basically an 'any pointer'). It's only C++ where this is an error (which is weird on its own, why even allow void pointers in C++ when the only reason they (presumably) exist doesn't work anymore).
The above code still doesn't compile in C because C complains about the function signatures being incompatible, even though the only difference is the argument type which itself is 'equivalent' in C - which I guess could be interepreted as an edge case in the C spec? (IMHO the function prototypes should actually be compatible).
"Compatibility" has a specific meaning in C which isn't the same as what you're describing here. What you're talking about is implicit conversions without narrowing.
Compatibility is essentially about ABI: https://en.cppreference.com/w/c/language/type#Compatible_typ...
The C standard is quite loose about pointer requirements. Conversions between data pointers or between function pointers are allowed, but conversions between each other are not guaranteed. You can read the nitty gritty here: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf
POSIX defines stricter requirements for pointers than the C standard. All pointers are compatible, so they have the same size and representation, and conversions between function pointers and void* are explicitly allowed.
Not all pointers. Function pointers, though often used that way, should not be cast to void. ISO C forbids it. (Implementations may allow it.)
ISO C forbids it, but, FWIW, POSIX requires it. So pick your standard I guess.
you can convert a short to an int without casting and it will roundtrip without loss of data; similarly you can roundtrip any data pointer through a void pointer without loss of data. It doesn't mean that an int has the same representation of a short or a void* has (necessarily) the same representation as any other pointer.
In C++ any pointer is also implicitly convertible to a void *, it is the reverse implicit conversion that is prohibited as it is not safe in the general case.
For consistency C++ should also prohibit implicit narrowing conversions (e.g. int to short ); I guess this was thought to break too much existing code and it is generally safer than a pointer conversion (although potentially lossy the behavior is fully defined). Many compilers will warn about it, and narrowing conversions are now prohibited in braced initialization.
Yes. Simply put, void pointers and struct pointers (the actual breaking example in Curl) are not guaranteed to have the same byte-level representation, by the C standard. (For example, a struct pointer may always be aligned and therefore use fewer bits than a void pointer.) Passing one to a function that expects the other may result in arbitrary breakage.
This is one of the very rare cases in C where something is technically Undefined Behaviour, but in practice works and is recommended.
The typedef struct trick, is very common idiom that creates _more_ safety, not less. All reasonable compilers should (and do) support this. It is sad that the ISO standard is not in line with reality at all times. (I say that as a member of the wg14 and the UB study group)
I Recommend Daniel keep his typedef struct definition, and then have an ifdef to revert to the void definition for when Clang does its UB sanitizer. While checking for prototype discrepancies is a very good thing to automate, Clang should add an exception for this.
There is no problem with using the typedef struct trick. The problem is assuming that such pointers have the same underlying representation as void pointers. Structs typically having stricter alignment requirements (whereas void* must be able to point to any byte) is one reason why their representation may differ from void*. This is why only [[un]signed] char* is guaranteed to have the same representation as void*.
Given:
This is perfectly fine: This is not:Alignment only matters when you use a pointer. This is a question of if technically non compatible signatures are UB by them selves. Unaligned access is UB for sure, but even if everything is perfectly aligned they may still be UB according to the standard.
Example:
Lets say you have a function that takes a pointer to a 32 bit integer.
void my_function(int x)
But in an other file you declare it as a function taking a 32 bit float, and call it:
extern void my_function(float x); ... my_function(&my_float);
This is very likely to work in almost all implementations. float and int have the same size and same alignment requirements, but the standard may still say its UB. So it is technically UB, but may be relied upon in practice.
> I say that as a member of the wg14 and the UB study group
Since you have experience in this area, do you know how likely it is that something like this could be resolved? I.e., if someone proposed "just make this defined", how likely would the C standards body be to agree and do so?
These things can be fixed and often are. Some one just needs to write a paper proposing a change. I might in fact do so on this issue. It should be resolvable.
> very rare cases in C where something is technically Undefined Behaviour, but in practice works and is recommended.
In the bad old days, such cases were very common. I think complaining about it is a part of the process that slowly resolves it.
Lots of code has been broken because people have written UB code and then optimzers become smarter and vreak the code.
This category of we call ”usless UB” are things that wont help optimizeayions, and all major compiler have to support beacause if they dont, their users complain too much, can be relied on.
Unfortunately its very hard for the average use to know what UB can be relied on (there isnt much).
Awesome writeup. Always interesting to read what Daniel has to say.
I think the fact that it turned out that he was wrong (and UBsan was right, as usual) is a great testament to the shortcomings of C.
Lots of people - both inexperienced and very experienced - celebrate it for being "simple" and "close to the hardware", but the truth of the matter is that it is precisely not close enough to the hardware for people who _know_ what the hardware is doing to be able to do what they expect, and it's too close to the hardware to be able to be able to ignore it.
Lots of experienced C programmers (and - guilt by association - C++ programmers as well) run into UB because they have clear expectations of the compiler. I.e., they know what the compiler should generate, more or less, and C is just a convenient notation. But compilers don't live up to those expectations, because they don't actually compile your code for the hardware. They compile it to the virtual machine abstraction defined by the standard, which very often works differently from any real architecture, and then translate that into machine code. Even though there is basically a single set of semantics that every single "relevant" (mainstream) architecture implements. This is a holdover from when C had to target architectures that are 100% irrelevant today.
Everybody's favorite example is signed integer overflow. In both x86-64 and ARM64, that just works - two's complement is the only relevant implementation, so there's no issue. But `int` in C and C++ is not that.
Almost every single common UB pitfall has reasonable behavior at the assembler level for every mainstream architecture, and almost every single niche architecture.
C gives you the illusion of being close to the hardware, but in actual reality the hardware is several steps removed, so if you want to leverage your knowledge of the hardware, calling conventions, assembly, or other low-level details, you have to go out of your way to work around the C standard.
(Aside: We need new languages to tackle this, and I coincidentally happen to like Rust. Lots of people coming from C or C++ are irritated and frustrated by Rust, but 99% of the time it's because Rust gives you a compile error where C would give you UB. This is one example of that out of thousands.)
Many of such issue were also fixed in languages like Ada and Modula-2, among others, which C and C++ folks used to call "programming with a straighjacket".
Many folks will complain about anything that breaks their beloved illusion that C is a fancier macro assembler, except even proper macro assemblers have less UB than regular C.
Being close to the hardware isn't a dogma. The point of being close to the hardware is to be able to write code that is (nearly) as perfomant as it could be if you used assembler. Undefined behavior that let's the compiler better reason about your code like signed integer overflow is entirely compatible with that goal. If anything, unsigned overflow should also be UB unless you explicitly tell the compiler that an operation should wrap, which really is not something you want in the common case where any overflow already means you have a bug.
Ada, Modula-2, PL/I, Mesa, NEWP, PL/S, Object Pascal, and many others are just as close with much less UB.
Wouldn't a faster and easier solution for existing code be to come up with a standard set of behaviours based on common architectures and make that a flag that can be switched on so expectations can be met (and call it "expected-C" or for the pun "ocean")?
AFAIK the reason for signed overflow still being UB isn't any exotic hardware property but because it enables some specific optimizations. Whether those optimizations are worth the trouble is up for discussion I guess.
No, it used to be the case that there were architectures where signed integers were represented as 1s complement, so portable code could not rely on signed integer overflow wrapping around (there would either be 2 bit patterns representing zero, or sometimes the all-ones pattern had special meaning, like a trap).
Using this type of UB is a "relatively" new thing (GCC started doing it in the 00s, which broke a lot of stuff in the Linux world, IIRC).
It _is_ true that somebody did the research (can't find the source right now) and found that defining signed integer overflow as wrapping did indeed make some code run slower. I'm skeptical that it matters.
That's why I wrote "still". AFAIK both C++ and C now expect integers to be in 2s complement and made unsigned overflow 'defined', but at the same time kept signed integer overflow as undefined behaviour.
unsigned overflow was always defined
C does give a lot of low-level control, but it is still an abstraction - as it should be.
Signed integer overflow is something were I personally like that it is UB, because I can turn on UBSan and find overflow bugs. If it were defined to wrap-around I could not do this.
Does anyone know of languages that achieve this? I'm interested, I'm currently implementing a project in x86 assembly for this reason, and am happy to try a higher level language.
Rust for example has virtually no UB (edit: no UB) in the safe subset, and it’s rare in most applications to have to use unsafe.
Just to nitpick, it's not "virtually" no UB, it's literally no UB.
(Barring compiler bugs, of course.)
And issues in the unsafe parts, which could cause UB in the safe part.
You are right, thanks for the correction.
Sorry, I meant a "close to the hardware" language re parent.
Depending on what you mean exactly I think Rust can be considered pretty close to the hardware. I.e. there's usually an obvious straightforward translation of each line of source code to an implementation in assembly language (which may not be what's actually emitted in practice due to optimizations, but the same is true in C).
There are of course some higher-level features like trait-based generics, so it's not really as close to the machine as C, but it's a lot closer to C IMO than something like Java (or even C++).
Ada, Modula-2 (both available on GCC), D (available on its reference implementation, GCC and LLVM), Swift, Object Pascal (Free Pascal, Delphi, Oxygen), Oberon variants (Oberon, Oberon-2, Oberon-07, Active Oberon, Component Pascal), Zig, Nim, Odin,...
Zig definitely has undefined behavior
https://ziglang.org/documentation/0.13.0/#Undefined-Behavior
All systems programming languages have undefined behaviour, the golden question is how much.
Also it wasn't what the parent asked for, and I quote: "close to the hardware" language
Rust is not (much?) further from the hardware than C++
C++ is further than rust in my opinion. Vtables that support inheritance, as well as stack unwinding for exceptions, are pretty complicated and totally implicit in C++. Okay rust also technically has unwinding for panics to be fair but it’s rather unusual in practice for programmers to use panics to mean anything other than “crash the program now”.
Like trait implementations, and trait objects.
That’s fair, trait objects do cause a table to be generated, but they don’t support inheritance and subjectively I think they’re used less often than the OOP features of c++ that lead to vtable-based dynamic dispatch. (Traits are extremely common in rust, but dyn trait objects somewhat less so)
Doesn't matter how common, the feature is there, and although we aren't yet that far, might even differ across implementations.
Also trait inheritance exists, enforced via trait bounds, what Rust doesn't support is class inheritance.
> Doesn't matter how common
Yes it does, if your concern is “how often do I encounter code where I can’t predict or control what it actually does on the hardware”.
Any time there is a compiler implementation that has to provide support for translating such features into machine code.
For signed overflow it's fascinating, Herb Sutter (WG21 convenor, Microsoft employee) writes that checking would "incur unacceptable costs".
Now, C++ programmers, were you consulted about these costs? Herb insists they're "unacceptable" but he provides no further information as to what the cost actually was, or who decided whether that cost was acceptable, much less how this could generalize across a wide variety of domains and platforms.
What's Herb's answer? You might hope that Herb would say OK, we'll provide wrapping for these types by default, it's not checked arithmetic but at least it's not UB. Nope.
C & C++ lack a particular sort of behavior that could solve a bunch of problems.
They've got undefined behavior: The standard imposes no requirements. Anything can happen. Assume the worst.
They've got unspecified behavior: Only covers behavior caused by an unspecified value or where the standard provides multiple choices, where the implementation need not document which choices are made in which situations.
They've got implementation-defined behavior: Unspecified behavior that the implementation must document.
They don't have a category for "Undefined behavior but the implementation must document". A lot of what is currently undefined behavior could better be put into this category, if it existed.
C++ 26 will (almost certainly, it's in the draft) add Erroneous Behaviour.
EB is well defined but definitely wrong, so it's a way for the standard to say:
Do not allow this to happen, but if you do, the consequence is definitely that.
The specific EB in the C++ 26 draft is the value of default uninitialized primitives. So e.g. int k; std::cout << k << "\n";
In C++ 23 and previous versions that's Undefined Behaviour, maybe it prints the lyrics to the National Anthem of the country where the compiler ran? Maybe it deletes all your files. But in C++ 26 the Erroneous Behaviour is that there's some integer value k, which your compiler vendor knew (and might tell you or even let you change it) and it prints that value, but you're naughty because this is definitively an error when it happens.
With GCC/clang can just add checking with -fsanitize=signed-integer-overflow -fsanitize-undefined-trap-on-error.
For my main software project, which is some numerical software for magnetic resonance imaging, this adds 12212 checks and the optimizer reduces them down to 3803. But I haven't done benchmarking yet, but I would guess that for most software it would not matter.
Basically the behavior is hardware-dependent, and nobody wants to mandate that C++ compilers generate a ton of extra instructions on hardware which does not behave a particular way.
Of course you can define your own checked integer types, using inline assembly to check the overflow flag where available.
Just so we're clear, yes, it's "hardware-dependent", but literally every single architecture and CPU model does the same reasonable thing, which is to wrap into the negative.
Any architecture that doesn't use 2s complement is so esoteric by now that it does not make any sense for a general-purpose C compiler to pretend they exist.
I think this is more of an edge case than you give it credit for.
(External linkage) functions, and their callsites, are quite special as they straddle the boundary between actual machine and virtual machine.
The calling convention makes fairly strict requirements of what that must compile down to, since caller and callee could be dynamically linked objects compiled by different compilers or even entirely different languages.
If you define a function that takes a void* parameter it will read a pointer from the RDI(?). If you pass a char* it will pass the pointer in that same register.
Now it must be said that if the compiler can see both the definition and call, then it's free to do whatever it wants e.g. inline or something.
But yeah it's a bit complicated?
Sure, I mean, my objection is with the pretense that a reasonable calling convention in 2024 could decide to represent void* differently from any other T. It makes total sense that C programmers expect pointers to be (transitively) convertible to/from void, so the fact that they aren't convertible in this way without a trampoline means that the standard contains surprises for even very experienced developers.
I postulate that almost all UB in the wild comes from the Standard diverging from (often very reasonable) expectations, and I see that as a big problem with the standard, at least as long as compilers can't reliably detect the problem at compile-time. (And yes, C++ is even more problematic here.)
I think one of the reasons Zig exists is that it contains far fewer surprises. The reason Rust exists is that it does a much better job at preventing and containing such surprises.
Every C compiler I ever used will tell you that void()(void) is not convertible to void ()(char). If people still do it then they are a bit on their own. But how is this then different to Rust's "unsafe"? (of course, there is other UB compiler do not tell you about, that this seems a bad example)
I would think the proper way to do this would be
If BUILDING_LIBCURL isn’t defined that tells code “CURL is identical to a struct named Curl_easy”. If it is defined, that also tells code what fields it has.Interesting problem. The typical solution in C++ to deal with type erasing function pointer types is to go through a trampoline function:
In plain C there is no way to generate the trampoline at the point of use in the same way template instantiation works, but it can be generated by a macro at global scope.Yes, it's very much undefined behavior. As I recall, GTK's glib does this all over the place for signals.
edit: I'm not advocating that this being UB is fine. I don't expect compilers to exploit this for optimization, because so many projects rely on this working. There might be room to extend compatible function types to make this defined.
> In 2016 I wanted to change the type universally to just typedef struct Curl_easy CURL; … as I thought we could do that without breaking neither API nor ABI.
This seems to be the obvious solution and how most libraries define their opaque handles. It doesn't break a guaranteed API and it doesn't break the ABI any more than than only using it when building the library - and you can check that it doesn't break the ABI on any platform where you want to guarantee ABI stability.
In the real world, we often want to avoid breaking people's code even when people rely on something that's not guaranteed by the API docs. It seems like Daniel's goal isn't to only be API-compatible in a technical sense (namely that perfectly written code which carefully avoids using anything in a way that's not explicitly guaranteed to work), but rather to avoid breaking people's existing code. I can respect that.
Yes but that is always a balancing act if you want to make any change at all since users can theoretically depend on any possible implementation detail or even on outright bugs.
Correct. It's always a balancing act. That means hard rules like "we will do any change which doesn't technically break any promises explicitly made in the documentation in a patch release" aren't appropriate, it's always a value judgement (and sometimes you get it wrong and revert the change once you realize that people were affected more than expected, as happened in the case of curl).
Ok, this is UB, calling a function pointer through a different type than its definition is a pretty clear example of UB. The problem here is that there's a confusion between "void * and char * are implicitly convertible in C" and "void * and char * are the same". The latter is true for many platforms (especially older ones) but not all (I think there were platforms where functionally they had `typedef char void`). There's a side note of conflating "this has defined behavior on my platform that is stable and works for me" and "it's not UB if it's stable and works on a platform", just like integer overflow is UB despite being entirely defined behavior on every platform under the sun.
Anyway, if folk are curious there are many platforms where not only can the representation of `void()(void)` and `void()(char)` be different - even if pointing to the same function - but the representation of even just the data pointers void* and char* may not be the same, again while pointing to the same memory.
For example, on platforms with pointer authentication function pointers are generally (I would say "always" but in principle it can be avoided) signed, and in some configuration the type of the function is incorporated into the function. Calling the function pointer requires authenticating the pointer, and authenticating the pointer requires that the call site agrees on the type of the pointer because otherwise the signature fails.
Absent actual pointer auth hardware you could imagine someone implementing this as some kind of monstrosity like this (very hypothetical, unpleasant, and footgun heavy) horror:
and you can immediately see that if you had code that used these but disagreed on the type of the function you'd have a bad time. With compiler+hardware pointer auth this is just handled transparently.In principle a pointer auth environment could apply this type discrimination logic to data pointers as well, but I'm unaware of any that do so implicitly. But if a platform did do so, then the incorrect type of the parameter would mean you would fail inside the function, if you were able to call it (say if you weren't using a function pointer, but had mistyped the prototype).
Similarly, in other environments, the pointer may be directly aware the type being referenced, and I believe that CHERI supports this, in which case even if you could call the function pointer when attempting to read the pointer I believe it would fail.
Having got here, you might be saying "but hang on C says void* and char* are the same", and we go all the way back to my first sentence where I said "are implicitly convertible" :D
On plenty of systems casting from one pointer type to another is an entirely source level feature and once lowered is completely invisible. But in the environments we're discussing
Under pointer auth it requires re-signing the pointer (you have to auth the original value to verify it, and then sign the result according to the new schema), and under CHERI I believe there are instructions for controlling how a pointer is tagged.But the important thing is the C does not say they are the same thing, just that the conversion is automatic, just like numbers and bools, or numbers and bools in JS, or numbers and strings in JS, or objects and strings in JS, or nothing and strings in JS, or .... :D
You are confusing "the same" with "compatible" and maybe also with "have the same alignment and representation".
Sorry I’m not clear what you’re saying - because I don’t know exactly which sentence(s) in my giant wall of text you’re referring to :D
I did try to simplify things as well rather than focusing on specific language technicalities, because just going into arbitrary amounts of technical detail didn’t seem like a good way to explain things, and I’ve found can result in people thinking you’re focusing on those “technicalities” rather than “real” issues, without understanding the technicalities are the root of the “real” issues.
Man rants about not expecting weird type system abuse that works on his machine to be undefined behavior
This is not a rant, but a well laid-out description of something a widely used software ran into as a result of a change in the compiler they use.
The #ifdef they had in place was a bit hackish, but I wouldn't call it abuse. It is a typical construction for C to do stuff like that which in general isn't without risk, but they have used it without any problems for years. The whole point of the blog post is to start a discussion whether sth. like this should rightfully be flagged as "unexpected behavior" or not.
It's not well laid out. The examples are malformed/illegal and the ifdef thing is stupid.
The author admits to not being a C undefined behavior expert and yet acts like they might know better than a tool made by such experts.
Looking up the rules and verifying the shown snippets takes at most 30 minutes at a leisurely pace, the author could have saved themselves the embarrassment.
I'm not going to write a blog post about how I didn't expect a color spectrometer pointed at the sky to say "BLUE" because I thought it might have been purple, "although I'm not an expert in wave lengths".
> The author admits to not being a C undefined behavior expert
At this stage, I would seriously doubt the credentials of anyone who claims to be a C undefined behaviour expert. Saying "I'm not a C UB expert" is just a realistic acknowledgement that UB is hard and we will get it wrong at some point without realising. The approach of having an automated tool tell you when UB is present is very sensible.
The ifdef thing certainly isn't "stpuid". It's not good design, you wouldn't design it that way if you made libcurl from scratch today, but it makes sense as a solution to the problem of, "we can't change the type of CURL* in the public API, but internally, it ought to be defined as a pointer to a struct". If it was well-defined behavior, it would probably have been the best solution possible given the constraints.
And apparently worked the last 28 or so years on loads of other machines (I think the question would be where doesn't curl run)?
Casting function pointers like this can break Emscripten [1]. In libxml2, I fixed all these issues 7 years ago. It can be painful, but "it worked for 28 years" is not an excuse.
[1] https://emscripten.org/docs/porting/guidelines/function_poin...
I believe AIX C++ name mangling includes function argument type information (with CV qualifiers!) so this is a real-world case where this does actually break. I suspect curl does not compile with the C++ compiler though.
So does the Itanium ABI (which is what most people would think of as the normal/standard/usual C++ ABI):
But I'm struggling to understand how this would cause things to break.I guess only if you directly expose C++ APIs in DLLs, which is a bad idea anyway.
It's literally the opposite of "works on my machine" because it's in curl (which is most likely the single most widely deployed code base in the world).
How would you implement objects and inheritance in C without function casts? CPython certainly uses these casts.
dlysm() even relies on a (void *) cast that is not C standard compliant.
C is useless for certain applications with this "undefined behavior".
That is the thing, people keep treating it as a portable macro assembler, when it stopped being so decades ago, and those folks haven't yet got the memo.
They say Rust is much harder than C for... disallowing these kinds of things?
For the same reason python is seen as easier: guardrails and checks are just an impediment right?