Returning an enum in 32-bit FFI is broken

[alexcrichton]

Here's the output of a shell session on 32-bit linux which causes the failure.

[~/rust[uv-rewrite]] $ cat foo.c 
enum A {                                                                                                            
  B, C                                                                                                              
};                                                                                                                  

enum A foo() { return B; }                                                                                          
[~/rust[uv-rewrite]] $ gcc -m32 foo.c -c -o foo.o  
[~/rust[uv-rewrite]] $ ar rcs libfoo.a foo.o    
[~/rust[uv-rewrite]] $ cat foo.rs    
#[allow(cstack)];                                                                                                   

#[repr(C)]                                                                                                          
enum A {                                                                                                            
  B, C                                                                                                              
}                                                                                                                   

#[link_args = "-lfoo"]                                                                                              
extern {                                                                                                            
  fn foo() -> A;                                                                                                    
}                                                                                                                   

fn main() {                                                                                                         
  unsafe { foo(); }                                                                                                 
}                                                                                                                   
[~/rust[uv-rewrite]] $ ./i686-unknown-linux-gnu/stage2/bin/rustc foo.rs 
[~/rust[uv-rewrite]] $ ./foo   
zsh: segmentation fault (core dumped)  ./foo                                                                        

cc @jld

[alexcrichton]

The problem which I have seen is that the function foo corrupts the return address of main, so foo is successfully called, returns, main does it's thing, and then when main returns it's return address is bad leading to sadness.

[jld]

Looks like this is because all enums are represented by LLVM structs, and on i386 a struct is always returned by memory. (On amd64, return values use the same classification logic as arguments.)

Enums (and structs) currently always turn into LLVM named structs so that recursion can be broken with forward declaration, but a C-like enum (no fields) can't have that problem. So that could be fixed... but the interface between trans::type_of and trans::adt would need some work.

[nikomatsakis]

On Tue, Nov 05, 2013 at 08:52:20PM -0800, Jed Davis wrote:

Enums (and structs) currently always turn into LLVM named structs so
that recursion can be broken with forward declaration, but a C-like
enum (no fields) can't have that problem. So that could be
fixed... but the interface between trans::type_of and trans::adt
would need some work.

This is perhaps the right fix. We could also try to patch up the
32-bit ABI code -- but that stuff is pretty complex as is.

[jld]

Work in progress: https://github.com/jld/rust/compare/enum-unstruct

[jld]

It makes the example at the top of the issue not crash, and examining the -S output on a simple case it looks right. But it needs tests — do we have examples of tests that link against C code that's part of the test (rather than a standard library routine)?

[jld]

Somewhat related: #5347, the source of the change that uses LLVM-struct-ness to determine whether to treat the type as a struct for the purpose of the C ABI. (But also, as long as we're getting into history: I think the only reason that C-like enums had an extra struct wrapper in the first place is because it was easier to do it that way in the pre-adt.rs era.)