Const evaluation with huge arrays consumes all available memory

[belovdv]

Maybe, this is duplicate of #93215

Example code:

fn main() {
    let _ = &[0; 1 << 40];
    println!();
}

I expected to see this happen:

Compilation, regardless of results, consume limited amount of memory. Ideally this code should work.

Instead, this happened:

Running rustc src/main.rs on example code consumes memory, until gets Killed.

Meta

rustc --version --verbose:

rustc 1.71.1 (eb26296b5 2023-08-03)
binary: rustc
commit-hash: eb26296b556cef10fb713a38f3d16b9886080f26
commit-date: 2023-08-03
host: x86_64-unknown-linux-gnu
release: 1.71.1
LLVM version: 16.0.5

or

rustc 1.73.0-nightly (474709a9a 2023-08-03)
binary: rustc
commit-hash: 474709a9a2a74a8bcf0055fadb335d0ca0d2d939
commit-date: 2023-08-03
host: x86_64-unknown-linux-gnu
release: 1.73.0-nightly
LLVM version: 16.0.5

Backtrace from gdb

#0  __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:351
#1  0x00007ffff53df683 in <rustc_const_eval::interpret::memory::AllocRefMut<rustc_middle::mir::interpret::AllocId, ()>>::write_scalar ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#2  0x00007ffff543e5e4 in <rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::copy_op_no_validate ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#3  0x00007ffff53a9a32 in <rustc_const_eval::interpret::eval_context::InterpCx<rustc_const_eval::const_eval::machine::CompileTimeInterpreter>>::statement ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#4  0x00007ffff536bda6 in rustc_const_eval::const_eval::eval_queries::eval_to_allocation_raw_provider ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#5  0x00007ffff50251f7 in rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::eval_to_allocation_raw::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 16]>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#6  0x00007ffff5b02b0c in rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_middle::ty::ParamEnvAnd<rustc_middle::mir::interpret::GlobalId>, rustc_middle::query::erase::Erased<[u8; 16]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#7  0x00007ffff5b027cd in rustc_query_impl::query_impl::eval_to_allocation_raw::get_query_non_incr::__rust_end_short_backtrace ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#8  0x00007ffff536b38e in rustc_const_eval::const_eval::eval_queries::eval_to_allocation_raw_provider ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#9  0x00007ffff50251f7 in rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::eval_to_allocation_raw::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 16]>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#10 0x00007ffff5b02b0c in rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::DefaultCache<rustc_middle::ty::ParamEnvAnd<rustc_middle::mir::interpret::GlobalId>, rustc_middle::query::erase::Erased<[u8; 16]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#11 0x00007ffff5b027cd in rustc_query_impl::query_impl::eval_to_allocation_raw::get_query_non_incr::__rust_end_short_backtrace ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#12 0x00007ffff5fd5886 in <rustc_const_eval::interpret::eval_context::InterpCx<rustc_mir_transform::const_prop::ConstPropMachine>>::eval_mir_constant ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#13 0x00007ffff5fcc7dc in <rustc_mir_transform::const_prop_lint::ConstPropagator as rustc_middle::mir::visit::Visitor>::visit_basic_block_data ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#14 0x00007ffff5fc8eff in <rustc_mir_transform::const_prop_lint::ConstProp as rustc_mir_transform::pass_manager::MirLint>::run_lint ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#15 0x00007ffff55d5436 in _RNvCs7bEHG8lNZ4h_19rustc_mir_transform38mir_drops_elaborated_and_const_checked.llvm.12707411324951536808 ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#16 0x00007ffff4fea7ee in rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::mir_drops_elaborated_and_const_checked::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 8]>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#17 0x00007ffff4fea7be in <rustc_query_impl::query_impl::mir_drops_elaborated_and_const_checked::dynamic_query::{closure#2} as core::ops::function::FnOnce<(rustc_middle::ty::context::TyCtxt, rustc_span::def_id::LocalDefId)>>::call_once ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#18 0x00007ffff50a09fd in rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#19 0x00007ffff689fd21 in rustc_query_impl::query_impl::mir_drops_elaborated_and_const_checked::get_query_non_incr::__rust_end_short_backtrace ()
--Type <RET> for more, q to quit, c to continue without paging--
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#20 0x00007ffff63c6c5d in <rustc_session::session::Session>::time::<(), rustc_interface::passes::analysis::{closure#2}> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#21 0x00007ffff63c4e66 in rustc_interface::passes::analysis ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#22 0x00007ffff640a8fa in rustc_query_impl::plumbing::__rust_begin_short_backtrace::<rustc_query_impl::query_impl::analysis::dynamic_query::{closure#2}::{closure#0}, rustc_middle::query::erase::Erased<[u8; 1]>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#23 0x00007ffff640a8e9 in _RNvYNCNvNtNtCs9t0W4sfjaFR_16rustc_query_impl10query_impl8analysis13dynamic_querys0_0INtNtNtCscS4HM4tpJt3_4core3ops8function6FnOnceTNtNtNtCsh2FAgqFZnx3_12rustc_middle2ty7context6TyCtxtuEE9call_onceBa_.llvm.10073415029295238485 ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#24 0x00007ffff65c7fe8 in rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::SingleCache<rustc_middle::query::erase::Erased<[u8; 1]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#25 0x00007ffff65c7db9 in rustc_query_impl::query_impl::analysis::get_query_non_incr::__rust_end_short_backtrace ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#26 0x00007ffff6152392 in <rustc_interface::queries::QueryResult<&rustc_middle::ty::context::GlobalCtxt>>::enter::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure#4}> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#27 0x00007ffff6151475 in <rustc_interface::interface::Compiler>::enter::<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}, core::result::Result<core::option::Option<rustc_interface::queries::Linker>, rustc_span::ErrorGuaranteed>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#28 0x00007ffff614f5c4 in rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#29 0x00007ffff614f1a1 in <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#30 0x00007ffff614e796 in std::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>> ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#31 0x00007ffff67a64c5 in <<std::thread::Builder>::spawn_unchecked_<rustc_interface::util::run_in_thread_pool_with_globals<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} ()
   from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so
#32 0x00007ffff3d777b5 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1985
#33 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1985
#34 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#35 0x00007ffff3a94ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#36 0x00007ffff3b26a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[Noratrieb]

This is a very big constant, more memory than you have. How would you expect this to work?

[scottmcm]

Compilation, regardless of results, consume limited amount of memory.

Note that this, in general, cannot be true. You can write and evaluate const fns that will need arbitrarily-high amounts of memory and compute.

[workingjubilee]

There doesn't seem to be anything we can do here. Constants are evaluated by an interpreter, thus when you write them down, you actually allocate the memory and run a program at compiletime. That is, compiletime is just another runtime. We can make the interpreter more efficient on some cases but this is genuinely insoluble in the general case.

Closing.