Eval bug: Crash over tool calls in Qwen3 Coder

[allo-]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
version: 1 (9008328)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes

Models

Qwen3 Coder 30B-A3

Problem description & steps to reproduce

llama-server crashes when qwen3-cli attempts a tool call. I am not sure if the tool call syntax is right, but it still should not crash.

First Bad Commit

No response

Relevant log output

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56	../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: File or directory not found
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56	in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x00007f064d699668 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:49
warning: 49	./nptl/cancellation.c: File or directory not found
#2  0x00007f064d6996ad in __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75	in ./nptl/cancellation.c
#3  0x00007f064d704787 in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: File or directory not found
#4  0x00007f064df4218b in ggml_print_backtrace () from libggml-base.so
#5  0x00007f064df535df in ggml_uncaught_exception() () from libggml-base.so
#6  0x00007f064d8b344a in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f064d8a15e9 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f064d8b36c8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x000055ff4a1d7662 in string_diff(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .cold] ()
#10 0x000055ff4a2d00ce in common_chat_msg_diff::compute_diffs(common_chat_msg const&, common_chat_msg const&) ()
#11 0x000055ff4a25299a in server_slot::update_chat_msg(std::vector<common_chat_msg_diff, std::allocator<common_chat_msg_diff> >&) ()
#12 0x000055ff4a2531ad in server_context::send_partial_response(server_slot&, completion_token_output const&) ()
#13 0x000055ff4a2540fb in server_context::process_token(completion_token_output&, server_slot&) ()
#14 0x000055ff4a280a07 in server_context::update_slots() ()
#15 0x000055ff4a2279fd in server_queue::start_loop() ()
#16 0x000055ff4a1f028d in main ()
[Inferior 1 (process 326508) detached]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Invalid diff: 'I'll create a file named test.txt in the current directory.

<function=write_file' not found at start of 'I'll create a file named test.txt in the current directory.


'
Aborted (core dumped)

[TetrisBlack]

I can reproduce this with using Goose
Command used to run llama

docker run --name docker-llama-qwen3 --gpus all --init --rm -p ${PORT}:8080 -v /home/llama/fast-models:/models ghcr.io/ggml-org/llama.cpp:server-cuda
      -m /models/Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf
      --flash-attn --metrics -ts 24,4
      --jinja -ngl 99 --threads -1 --ctx-size 131072
      --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --repeat-penalty 1.05

error with version b6089:

libggml-base.so(+0x16ceb)[0x7f7d43df2ceb]
libggml-base.so(ggml_print_backtrace+0x21f)[0x7f7d43df314f]
libggml-base.so(+0x2859f)[0x7f7d43e0459f]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c)[0x7f7d43c5c20c]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xae277)[0x7f7d43c5c277]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xae4d8)[0x7f7d43c5c4d8]
libllama.so(+0x65e1d)[0x7f7d43eeae1d]
libllama.so(_Z25llama_grammar_accept_implR13llama_grammari+0x26f)[0x7f7d43f2ddcf]
/app/llama-server(+0x1ec19c)[0x6492d3f0019c]
/app/llama-server(+0xdea47)[0x6492d3df2a47]
/app/llama-server(+0x84f5d)[0x6492d3d98f5d]
/app/llama-server(+0x4ce55)[0x6492d3d60e55]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f7d438a7d90]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f7d438a7e40]
/app/llama-server(+0x4e8a5)[0x6492d3d628a5]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Unexpected empty grammar stack after accepting piece:  <function

Also tried with the fixed template that was uploaded by the unsloth team. Thread in Huggingface

---

Also, maybe related to this. After b6055 mistral-3.2 can't be used with tool calling.
error:

got exception: {"code":500,"message":"Trying to access property '0' on null! at row 243, column 16:\n        {%- else %}\n            {{- message['content'][0]['text'] }}\n               ^\n        {%- endif %}\n at row 243, column 16:\n        {%- else %}\n            {{- message['content'][0]['text'] }}\n               ^\n        {%- endif %}\n at row 243, column 13:\n        {%- else %}\n            {{- message['content'][0]['text'] }}\n            ^\n        {%- endif %}\n at row 242, column 20:\n            {{- message['content'] }}\n        {%- else %}\n                   ^\n            {{- message['content'][0]['text'] }}\n at row 240, column 9:\n    {%- elif message['role'] == 'assistant' %}\n        {%- if message['content'] is string %}\n        ^\n            {{- message['content'] }}\n at row 239, column 47:\n\n    {%- elif message['role'] == 'assistant' %}\n                                              ^\n        {%- if message['content'] is string %}\n at row 197, column 5:\n{%- for message in loop_messages %}\n    {%- if message['role'] == 'user' %}\n    ^\n\n at row 196, column 36:\n\n{%- for message in loop_messages %}\n                                   ^\n    {%- if message['role'] == 'user' %}\n at row 196, column 1:\n\n{%- for message in loop_messages %}\n^\n    {%- if message['role'] == 'user' %}\n at row 1, column 69:\n{#- Copyright 2025-present the Unsloth team. All rights reserved. #}\n                                                                    ^\n{#- Licensed under the Apache License, Version 2.0 (the \"License\") #}\n","type":"server_error"}

the same call works without problems in version b6055

[ochafik]

Hi @allo- , @TetrisBlack

Thanks for reporting this crash! It would really help if you managed to reproduce it w/ the --verbose flag and paste the full output (esp. the request bit, and the part where the model output is printed before causing the crash), as I haven't managed to repro yet.

There is likely a bug in the current parsing support that is exacerbated by Qwen3 Coder having a different output than the one currently supported. I've started looking at the Qwen3 Code chat template, its tool calling syntax is annoyingly not based on JSON and doesn't seem to have any support to escape outputs (e.g. if the model outputs some code that contains the literal </parameter>, it will cause the parsing to terminate), but I'll probably still try and add support for it.

[TetrisBlack]

@ochafik

I tried to reproduce it with the latest docker version and I couldn't reproduce the crash. It still errors out and throws an 500 error.

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-alderlake.so
build: 6097 (9515c613) with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
system info: n_threads = 20, n_threads_batch = 20, total_threads = 20

system_info: n_threads = 20 (n_threads_batch = 20) / 20 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 19
main: loading model
srv    load_model: loading model '/models/Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23861 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3060) - 7227 MiB free
llama_model_loader: loaded meta data with 42 key-value pairs and 579 tensors from /models/Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3-Coder-30B-A3B-Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen3-Coder-30B-A3B-Instruct
llama_model_loader: - kv   5:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   6:                         general.size_label str              = 30B-A3B
llama_model_loader: - kv   7:                            general.license str              = apache-2.0
llama_model_loader: - kv   8:                       general.license.link str              = https://huggingface.co/Qwen/Qwen3-Cod...
llama_model_loader: - kv   9:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv  10:                   general.base_model.count u32              = 1
llama_model_loader: - kv  11:                  general.base_model.0.name str              = Qwen3 Coder 30B A3B Instruct
llama_model_loader: - kv  12:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  13:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen3-Cod...
llama_model_loader: - kv  14:                               general.tags arr[str,2]       = ["unsloth", "text-generation"]
llama_model_loader: - kv  15:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  16:                    qwen3moe.context_length u32              = 262144
llama_model_loader: - kv  17:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  18:               qwen3moe.feed_forward_length u32              = 5472
llama_model_loader: - kv  19:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  20:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  21:                    qwen3moe.rope.freq_base f32              = 10000000.000000
llama_model_loader: - kv  22:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  23:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  24:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  25:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  26:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  27:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  28: qwen3moe.expert_shared_feed_forward_length u32              = 0
llama_model_loader: - kv  29:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  30:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  31:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  32:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  33:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  34:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  35:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  36:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  37:                    tokenizer.chat_template str              = {#- Copyright 2025-present the Unslot...
llama_model_loader: - kv  38:               general.quantization_version u32              = 2
llama_model_loader: - kv  39:                          general.file_type u32              = 15
llama_model_loader: - kv  40:                      quantize.imatrix.file str              = Qwen3-Coder-30B-A3B-Instruct-GGUF/ima...
llama_model_loader: - kv  41:             quantize.imatrix.entries_count u32              = 383
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q4_K:  292 tensors
llama_model_loader: - type q5_K:   35 tensors
llama_model_loader: - type q6_K:   11 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 16.45 GiB (4.63 BPW) 
init_tokenizer: initializing tokenizer for type 2
load: control token: 151660 '<|fim_middle|>' is not marked as EOG
load: control token: 151659 '<|fim_prefix|>' is not marked as EOG
load: control token: 151653 '<|vision_end|>' is not marked as EOG
load: control token: 151648 '<|box_start|>' is not marked as EOG
load: control token: 151646 '<|object_ref_start|>' is not marked as EOG
load: control token: 151649 '<|box_end|>' is not marked as EOG
load: control token: 151655 '<|image_pad|>' is not marked as EOG
load: control token: 151651 '<|quad_end|>' is not marked as EOG
load: control token: 151647 '<|object_ref_end|>' is not marked as EOG
load: control token: 151652 '<|vision_start|>' is not marked as EOG
load: control token: 151654 '<|vision_pad|>' is not marked as EOG
load: control token: 151656 '<|video_pad|>' is not marked as EOG
load: control token: 151644 '<|im_start|>' is not marked as EOG
load: control token: 151661 '<|fim_suffix|>' is not marked as EOG
load: control token: 151650 '<|quad_start|>' is not marked as EOG
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 26
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 262144
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 5472
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 10000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 262144
print_info: rope_finetuned   = unknown
print_info: model type       = 30B.A3B
print_info: model params     = 30.53 B
print_info: general.name     = Qwen3-Coder-30B-A3B-Instruct
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 11 ','
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151654 '<|vision_pad|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: layer   0 assigned to device CUDA0, is_swa = 0
load_tensors: layer   1 assigned to device CUDA0, is_swa = 0
load_tensors: layer   2 assigned to device CUDA0, is_swa = 0
load_tensors: layer   3 assigned to device CUDA0, is_swa = 0
load_tensors: layer   4 assigned to device CUDA0, is_swa = 0
load_tensors: layer   5 assigned to device CUDA0, is_swa = 0
load_tensors: layer   6 assigned to device CUDA0, is_swa = 0
load_tensors: layer   7 assigned to device CUDA0, is_swa = 0
load_tensors: layer   8 assigned to device CUDA0, is_swa = 0
load_tensors: layer   9 assigned to device CUDA0, is_swa = 0
load_tensors: layer  10 assigned to device CUDA0, is_swa = 0
load_tensors: layer  11 assigned to device CUDA0, is_swa = 0
load_tensors: layer  12 assigned to device CUDA0, is_swa = 0
load_tensors: layer  13 assigned to device CUDA0, is_swa = 0
load_tensors: layer  14 assigned to device CUDA0, is_swa = 0
load_tensors: layer  15 assigned to device CUDA0, is_swa = 0
load_tensors: layer  16 assigned to device CUDA0, is_swa = 0
load_tensors: layer  17 assigned to device CUDA0, is_swa = 0
load_tensors: layer  18 assigned to device CUDA0, is_swa = 0
load_tensors: layer  19 assigned to device CUDA0, is_swa = 0
load_tensors: layer  20 assigned to device CUDA0, is_swa = 0
load_tensors: layer  21 assigned to device CUDA0, is_swa = 0
load_tensors: layer  22 assigned to device CUDA0, is_swa = 0
load_tensors: layer  23 assigned to device CUDA0, is_swa = 0
load_tensors: layer  24 assigned to device CUDA0, is_swa = 0
load_tensors: layer  25 assigned to device CUDA0, is_swa = 0
load_tensors: layer  26 assigned to device CUDA0, is_swa = 0
load_tensors: layer  27 assigned to device CUDA0, is_swa = 0
load_tensors: layer  28 assigned to device CUDA0, is_swa = 0
load_tensors: layer  29 assigned to device CUDA0, is_swa = 0
load_tensors: layer  30 assigned to device CUDA0, is_swa = 0
load_tensors: layer  31 assigned to device CUDA0, is_swa = 0
load_tensors: layer  32 assigned to device CUDA0, is_swa = 0
load_tensors: layer  33 assigned to device CUDA0, is_swa = 0
load_tensors: layer  34 assigned to device CUDA0, is_swa = 0
load_tensors: layer  35 assigned to device CUDA0, is_swa = 0
load_tensors: layer  36 assigned to device CUDA0, is_swa = 0
load_tensors: layer  37 assigned to device CUDA0, is_swa = 0
load_tensors: layer  38 assigned to device CUDA0, is_swa = 0
load_tensors: layer  39 assigned to device CUDA0, is_swa = 0
load_tensors: layer  40 assigned to device CUDA0, is_swa = 0
load_tensors: layer  41 assigned to device CUDA0, is_swa = 0
load_tensors: layer  42 assigned to device CUDA1, is_swa = 0
load_tensors: layer  43 assigned to device CUDA1, is_swa = 0
load_tensors: layer  44 assigned to device CUDA1, is_swa = 0
load_tensors: layer  45 assigned to device CUDA1, is_swa = 0
load_tensors: layer  46 assigned to device CUDA1, is_swa = 0
load_tensors: layer  47 assigned to device CUDA1, is_swa = 0
load_tensors: layer  48 assigned to device CUDA1, is_swa = 0
load_tensors: tensor 'token_embd.weight' (q4_K) (and 0 others) cannot be used with preferred buffer type CUDA_Host, using CPU instead
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors:        CUDA0 model buffer size = 14286.90 MiB
load_tensors:        CUDA1 model buffer size =  2387.47 MiB
load_tensors:   CPU_Mapped model buffer size =   166.92 MiB
................................srv  log_server_r: request: GET /health 172.17.0.1 503
srv  log_server_r: request:  
srv  log_server_r: response: {"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}
................................................srv  log_server_r: request: GET /health 172.17.0.1 503
srv  log_server_r: request:  
srv  log_server_r: response: {"error":{"code":503,"message":"Loading model","type":"unavailable_error"}}
...................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 131072
llama_context: n_ctx_per_seq = 131072
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = 1
llama_context: kv_unified    = false
llama_context: freq_base     = 10000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
create_memory: n_ctx = 131072 (padded)
llama_kv_cache_unified: layer   0: dev = CUDA0
llama_kv_cache_unified: layer   1: dev = CUDA0
llama_kv_cache_unified: layer   2: dev = CUDA0
llama_kv_cache_unified: layer   3: dev = CUDA0
llama_kv_cache_unified: layer   4: dev = CUDA0
llama_kv_cache_unified: layer   5: dev = CUDA0
llama_kv_cache_unified: layer   6: dev = CUDA0
llama_kv_cache_unified: layer   7: dev = CUDA0
llama_kv_cache_unified: layer   8: dev = CUDA0
llama_kv_cache_unified: layer   9: dev = CUDA0
llama_kv_cache_unified: layer  10: dev = CUDA0
llama_kv_cache_unified: layer  11: dev = CUDA0
llama_kv_cache_unified: layer  12: dev = CUDA0
llama_kv_cache_unified: layer  13: dev = CUDA0
llama_kv_cache_unified: layer  14: dev = CUDA0
llama_kv_cache_unified: layer  15: dev = CUDA0
llama_kv_cache_unified: layer  16: dev = CUDA0
llama_kv_cache_unified: layer  17: dev = CUDA0
llama_kv_cache_unified: layer  18: dev = CUDA0
llama_kv_cache_unified: layer  19: dev = CUDA0
llama_kv_cache_unified: layer  20: dev = CUDA0
llama_kv_cache_unified: layer  21: dev = CUDA0
llama_kv_cache_unified: layer  22: dev = CUDA0
llama_kv_cache_unified: layer  23: dev = CUDA0
llama_kv_cache_unified: layer  24: dev = CUDA0
llama_kv_cache_unified: layer  25: dev = CUDA0
llama_kv_cache_unified: layer  26: dev = CUDA0
llama_kv_cache_unified: layer  27: dev = CUDA0
llama_kv_cache_unified: layer  28: dev = CUDA0
llama_kv_cache_unified: layer  29: dev = CUDA0
llama_kv_cache_unified: layer  30: dev = CUDA0
llama_kv_cache_unified: layer  31: dev = CUDA0
llama_kv_cache_unified: layer  32: dev = CUDA0
llama_kv_cache_unified: layer  33: dev = CUDA0
llama_kv_cache_unified: layer  34: dev = CUDA0
llama_kv_cache_unified: layer  35: dev = CUDA0
llama_kv_cache_unified: layer  36: dev = CUDA0
llama_kv_cache_unified: layer  37: dev = CUDA0
llama_kv_cache_unified: layer  38: dev = CUDA0
llama_kv_cache_unified: layer  39: dev = CUDA0
llama_kv_cache_unified: layer  40: dev = CUDA0
llama_kv_cache_unified: layer  41: dev = CUDA0
llama_kv_cache_unified: layer  42: dev = CUDA1
llama_kv_cache_unified: layer  43: dev = CUDA1
llama_kv_cache_unified: layer  44: dev = CUDA1
llama_kv_cache_unified: layer  45: dev = CUDA1
llama_kv_cache_unified: layer  46: dev = CUDA1
llama_kv_cache_unified: layer  47: dev = CUDA1
llama_kv_cache_unified:      CUDA0 KV buffer size =  5712.00 MiB
llama_kv_cache_unified:      CUDA1 KV buffer size =   816.00 MiB
llama_kv_cache_unified: size = 6528.00 MiB (131072 cells,  48 layers,  1/1 seqs), K (q8_0): 3264.00 MiB, V (q8_0): 3264.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 3
llama_context: max_nodes = 4632
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 0
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
graph_reserve: reserving a graph for ubatch with n_tokens =    1, n_seqs =  1, n_outputs =    1
graph_reserve: reserving a graph for ubatch with n_tokens =  512, n_seqs =  1, n_outputs =  512
llama_context:      CUDA0 compute buffer size =  1256.04 MiB
llama_context:      CUDA1 compute buffer size =   828.80 MiB
llama_context:  CUDA_Host compute buffer size =  1028.05 MiB
llama_context: graph nodes  = 2983
llama_context: graph splits = 3
clear_adapter_lora: call
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 131072
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
set_warmup: value = 1
set_warmup: value = 0
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 131072
slot        reset: id  0 | task -1 | 
main: model loaded
main: chat template, chat_template: {#- Copyright 2025-present the Unsloth team. All rights reserved. #}
{#- Licensed under the Apache License, Version 2.0 (the "License") #}
{#- Edits made by Unsloth to fix the chat template #}
{% macro render_item_list(item_list, tag_name='required') %}
    {%- if item_list is defined and item_list is iterable and item_list | length > 0 %}
        {%- if tag_name %}{{- '\n<' ~ tag_name ~ '>' -}}{% endif %}
            {{- '[' }}
                {%- for item in item_list -%}
                    {%- if loop.index > 1 %}{{- ", "}}{% endif -%}
                    {%- if item is string -%}
                        {{ "`" ~ item ~ "`" }}
                    {%- else -%}
                        {{ item }}
                    {%- endif -%}
                {%- endfor -%}
            {{- ']' }}
        {%- if tag_name %}{{- '</' ~ tag_name ~ '>' -}}{% endif %}
    {%- endif %}
{% endmacro %}

{%- if messages[0]["role"] == "system" %}
    {%- set system_message = messages[0]["content"] %}
    {%- set loop_messages = messages[1:] %}
{%- else %}
    {%- set loop_messages = messages %}
{%- endif %}

{%- if not tools is defined %}
    {%- set tools = [] %}
{%- endif %}

{%- if system_message is defined %}
    {{- "<|im_start|>system\n" + system_message }}
{%- else %}
    {%- if tools is iterable and tools | length > 0 %}
        {{- "<|im_start|>system\nYou are Qwen, a helpful AI assistant that can interact with a computer to solve tasks." }}
    {%- endif %}
{%- endif %}
{%- if tools is iterable and tools | length > 0 %}
    {{- "\n\nYou have access to the following functions:\n\n" }}
    {{- "<tools>" }}
    {%- for tool in tools %}
        {%- if tool.function is defined %}
            {%- set tool = tool.function %}
        {%- endif %}
        {{- "\n<function>\n<name>" ~ tool.name ~ "</name>" }}
        {{- '\n<description>' ~ (tool.description | trim) ~ '</description>' }}
        {{- '\n<parameters>' }}
        {%- for param_name, param_fields in tool.parameters.properties|items %}
            {{- '\n<parameter>' }}
            {{- '\n<name>' ~ param_name ~ '</name>' }}
            {%- if param_fields.type is defined %}
                {{- '\n<type>' ~ (param_fields.type | string) ~ '</type>' }}
            {%- endif %}
            {%- if param_fields.description is defined %}
                {{- '\n<description>' ~ (param_fields.description | trim) ~ '</description>' }}
            {%- endif %}
            {{- render_item_list(param_fields.enum, 'enum') }}
            {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}
            {%- for json_key in param_fields %}
                {%- if json_key not in handled_keys %}
                    {%- set normed_json_key = json_key | replace("-", "_") | replace(" ", "_") | replace("$", "") %}
                    {%- if param_fields[json_key] is mapping %}
                        {{- '\n<' ~ normed_json_key ~ '>' ~ (param_fields[json_key] | tojson | safe) ~ '</' ~ normed_json_key ~ '>' }}
                    {%- else %}
                        {{- '\n<' ~ normed_json_key ~ '>' ~ (param_fields[json_key] | string) ~ '</' ~ normed_json_key ~ '>' }}
                    {%- endif %}
                {%- endif %}
            {%- endfor %}
            {{- render_item_list(param_fields.required, 'required') }}
            {{- '\n</parameter>' }}
        {%- endfor %}
        {{- render_item_list(tool.parameters.required, 'required') }}
        {{- '\n</parameters>' }}
        {%- if tool.return is defined %}
            {%- if tool.return is mapping %}
                {{- '\n<return>' ~ (tool.return | tojson | safe) ~ '</return>' }}
            {%- else %}
                {{- '\n<return>' ~ (tool.return | string) ~ '</return>' }}
            {%- endif %}
        {%- endif %}
        {{- '\n</function>' }}
    {%- endfor %}
    {{- "\n</tools>" }}
    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
{%- endif %}
{%- if system_message is defined %}
    {{- '<|im_end|>\n' }}
{%- else %}
    {%- if tools is iterable and tools | length > 0 %}
        {{- '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in loop_messages %}
    {%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
        {{- '<|im_start|>' + message.role }}
        {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
            {{- '\n' + message.content | trim + '\n' }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
            {%- if tool_call.arguments is defined %}
                {%- for args_name, args_value in tool_call.arguments|items %}
                    {{- '<parameter=' + args_name + '>\n' }}
                    {%- set args_value = args_value if args_value is string else args_value | string %}
                    {{- args_value }}
                    {{- '\n</parameter>\n' }}
                {%- endfor %}
            {%- endif %}
            {{- '</function>\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "user" or message.role == "system" or message.role == "assistant" %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "tool" %}
        {%- if loop.previtem and loop.previtem.role != "tool" %}
            {{- '<|im_start|>user\n' }}
        {%- endif %}
        {{- '<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>\n' }}
        {%- if not loop.last and loop.nextitem.role != "tool" %}
            {{- '<|im_end|>\n' }}
        {%- elif loop.last %}
            {{- '<|im_end|>\n' }}
        {%- endif %}
    {%- else %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}
{#- Copyright 2025-present the Unsloth team. All rights reserved. #}
{#- Licensed under the Apache License, Version 2.0 (the "License") #}, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
que    start_loop: processing new tasks
que    start_loop: update slots
srv  update_slots: all slots are idle
srv  kv_cache_cle: clearing KV cache
que    start_loop: waiting for new tasks
srv  log_server_r: request: GET /health 172.17.0.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"status":"ok"}
request: {"stream": true, "model": "qwen3-30b-coder-2507", "messages": [{"role": "system", "content": "User Context:\n\n"}, {"role": "user", "content": "Was f\u00fcr spiele sind gestern rausgekommen?"}], "tools": [{"type": "function", "function": {"type": "function", "name": "tool_brave_image_search_post", "description": "A tool for searching the web for images using the Brave Search API.", "parameters": {"type": "object", "properties": {"searchTerm": {"type": "string", "title": "Searchterm", "description": "The term to search the internet for images of"}, "count": {"type": "number", "title": "Count", "description": "The number of images to search for, minimum 1, maximum 3", "default": 1}}, "required": ["searchTerm"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_web_search_post", "description": "Performs a web search using the Brave Search API, ideal for general queries, and online content. Use this for broad information gathering, recent events, or when you need diverse web sources. Maximum 20 results per request", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "offset": {"type": "number", "title": "Offset", "description": "The offset for pagination, minimum 0", "default": 0}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_local_search_post", "description": "Searches for local businesses and places using Brave's Local Search API. Best for queries related to physical locations, businesses, restaurants, services, etc. Returns detailed information including:\n- Business names and addresses\n- Ratings and review counts\n- Phone numbers and opening hours\nUse this when the query implies 'near me' or mentions specific locations. Automatically falls back to web search if no local results are found.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "Local search query (e.g. 'pizza near Central Park')"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_news_search_post", "description": "Searches for news articles using the Brave Search API. Use this for recent events, trending topics, or specific news stories. Returns a list of articles with titles, URLs, and descriptions. Maximum 20 results per request.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for news articles, trending topics, or recent events"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_video_search_post", "description": "Searches for videos using the Brave Search API. Use this for video content, tutorials, or any media-related queries. Returns a list of videos with titles, URLs, and descriptions. Maximum 20 results per request.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for videos of"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}]}
got exception: {"code":500,"message":"Value is not callable: null at row 62, column 114:\n                {%- if json_key not in handled_keys %}\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n                                                                                                                 ^\n                    {%- if param_fields[json_key] is mapping %}\n at row 62, column 21:\n                {%- if json_key not in handled_keys %}\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n                    ^\n                    {%- if param_fields[json_key] is mapping %}\n at row 61, column 55:\n            {%- for json_key in param_fields %}\n                {%- if json_key not in handled_keys %}\n                                                      ^\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 61, column 17:\n            {%- for json_key in param_fields %}\n                {%- if json_key not in handled_keys %}\n                ^\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 60, column 48:\n            {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n            {%- for json_key in param_fields %}\n                                               ^\n                {%- if json_key not in handled_keys %}\n at row 60, column 13:\n            {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n            {%- for json_key in param_fields %}\n            ^\n                {%- if json_key not in handled_keys %}\n at row 49, column 80:\n        {{- '\\n<parameters>' }}\n        {%- for param_name, param_fields in tool.parameters.properties|items %}\n                                                                               ^\n            {{- '\\n<parameter>' }}\n at row 49, column 9:\n        {{- '\\n<parameters>' }}\n        {%- for param_name, param_fields in tool.parameters.properties|items %}\n        ^\n            {{- '\\n<parameter>' }}\n at row 42, column 29:\n    {{- \"<tools>\" }}\n    {%- for tool in tools %}\n                            ^\n        {%- if tool.function is defined %}\n at row 42, column 5:\n    {{- \"<tools>\" }}\n    {%- for tool in tools %}\n    ^\n        {%- if tool.function is defined %}\n at row 39, column 51:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n                                                  ^\n    {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 39, column 1:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n^\n    {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 1, column 69:\n{#- Copyright 2025-present the Unsloth team. All rights reserved. #}\n                                                                    ^\n{#- Licensed under the Apache License, Version 2.0 (the \"License\") #}\n","type":"server_error"}
srv  log_server_r: request: POST /v1/chat/completions 172.17.0.1 500
srv  log_server_r: request:  {"stream": true, "model": "qwen3-30b-coder-2507", "messages": [{"role": "system", "content": "User Context:\n\n"}, {"role": "user", "content": "Was f\u00fcr spiele sind gestern rausgekommen?"}], "tools": [{"type": "function", "function": {"type": "function", "name": "tool_brave_image_search_post", "description": "A tool for searching the web for images using the Brave Search API.", "parameters": {"type": "object", "properties": {"searchTerm": {"type": "string", "title": "Searchterm", "description": "The term to search the internet for images of"}, "count": {"type": "number", "title": "Count", "description": "The number of images to search for, minimum 1, maximum 3", "default": 1}}, "required": ["searchTerm"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_web_search_post", "description": "Performs a web search using the Brave Search API, ideal for general queries, and online content. Use this for broad information gathering, recent events, or when you need diverse web sources. Maximum 20 results per request", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "offset": {"type": "number", "title": "Offset", "description": "The offset for pagination, minimum 0", "default": 0}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_local_search_post", "description": "Searches for local businesses and places using Brave's Local Search API. Best for queries related to physical locations, businesses, restaurants, services, etc. Returns detailed information including:\n- Business names and addresses\n- Ratings and review counts\n- Phone numbers and opening hours\nUse this when the query implies 'near me' or mentions specific locations. Automatically falls back to web search if no local results are found.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "Local search query (e.g. 'pizza near Central Park')"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_news_search_post", "description": "Searches for news articles using the Brave Search API. Use this for recent events, trending topics, or specific news stories. Returns a list of articles with titles, URLs, and descriptions. Maximum 20 results per request.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for news articles, trending topics, or recent events"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}, {"type": "function", "function": {"type": "function", "name": "tool_brave_video_search_post", "description": "Searches for videos using the Brave Search API. Use this for video content, tutorials, or any media-related queries. Returns a list of videos with titles, URLs, and descriptions. Maximum 20 results per request.", "parameters": {"type": "object", "properties": {"query": {"type": "string", "title": "Query", "description": "The term to search the internet for videos of"}, "count": {"type": "number", "title": "Count", "description": "The number of results to return, minimum 1, maximum 20", "default": 10}, "freshness": {"type": "string", "title": "Freshness", "description": "Filters search results by when they were discovered.\nThe following values are supported:\n- pd: Discovered within the last 24 hours.\n- pw: Discovered within the last 7 Days.\n- pm: Discovered within the last 31 Days.\n- py: Discovered within the last 365 Days.\n- YYYY-MM-DDtoYYYY-MM-DD: Custom date range (e.g., 2022-04-01to2022-07-30)"}}, "required": ["query"]}}}]}
srv  log_server_r: response: {"error":{"code":500,"message":"Value is not callable: null at row 62, column 114:\n                {%- if json_key not in handled_keys %}\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n                                                                                                                 ^\n                    {%- if param_fields[json_key] is mapping %}\n at row 62, column 21:\n                {%- if json_key not in handled_keys %}\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n                    ^\n                    {%- if param_fields[json_key] is mapping %}\n at row 61, column 55:\n            {%- for json_key in param_fields %}\n                {%- if json_key not in handled_keys %}\n                                                      ^\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 61, column 17:\n            {%- for json_key in param_fields %}\n                {%- if json_key not in handled_keys %}\n                ^\n                    {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 60, column 48:\n            {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n            {%- for json_key in param_fields %}\n                                               ^\n                {%- if json_key not in handled_keys %}\n at row 60, column 13:\n            {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n            {%- for json_key in param_fields %}\n            ^\n                {%- if json_key not in handled_keys %}\n at row 49, column 80:\n        {{- '\\n<parameters>' }}\n        {%- for param_name, param_fields in tool.parameters.properties|items %}\n                                                                               ^\n            {{- '\\n<parameter>' }}\n at row 49, column 9:\n        {{- '\\n<parameters>' }}\n        {%- for param_name, param_fields in tool.parameters.properties|items %}\n        ^\n            {{- '\\n<parameter>' }}\n at row 42, column 29:\n    {{- \"<tools>\" }}\n    {%- for tool in tools %}\n                            ^\n        {%- if tool.function is defined %}\n at row 42, column 5:\n    {{- \"<tools>\" }}\n    {%- for tool in tools %}\n    ^\n        {%- if tool.function is defined %}\n at row 39, column 51:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n                                                  ^\n    {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 39, column 1:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n^\n    {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 1, column 69:\n{#- Copyright 2025-present the Unsloth team. All rights reserved. #}\n                                                                    ^\n{#- Licensed under the Apache License, Version 2.0 (the \"License\") #}\n","type":"server_error"}}
srv  log_server_r: request: GET /health 127.0.0.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"status":"ok"}
srv  log_server_r: request: GET /health 127.0.0.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"status":"ok"}
srv  log_server_r: request: GET /health 127.0.0.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"status":"ok"}
srv  log_server_r: request: GET /health 127.0.0.1 200
srv  log_server_r: request:  
srv  log_server_r: response: {"status":"ok"}

---

Its now the same or similar error message as on the mistral models while trying to toolcall. Maybe I need to create a new defect for Devstral and Mistral 3.2. I currently run the version b6055 on all tool call models. Where the mistral models are working with toolcalling, but qwen3 coder errors there as well. (see above)

[remon-nashid]

Gemma3 27B and Llama Scout have recently regressed as well. Witnessing a similar error to the ones above.

Gemma

got exception: {"code":500,"message":"Trying to access property '0' on null! at row 8, column 36:\n    {%- else -%}\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n                                   ^\n\n at row 8, column 36:\n    {%- else -%}\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n                                   ^\n\n at row 10, column 3:\n\n' -%}\n  ^\n    {%- endif -%}\n at row 8, column 9:\n    {%- else -%}\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n        ^\n\n at row 7, column 17:\n' -%}\n    {%- else -%}\n                ^\n        {%- set first_user_prefix = messages[0]['content'][0]['text'] + '\n at row 3, column 5:\n{%- if messages[0]['role'] == 'system' -%}\n    {%- if messages[0]['content'] is string -%}\n    ^\n        {%- set first_user_prefix = messages[0]['content'] + '\n at row 2, column 43:\n{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n                                          ^\n    {%- if messages[0]['content'] is string -%}\n at row 2, column 1:\n{{ bos_token }}\n{%- if messages[0]['role'] == 'system' -%}\n^\n    {%- if messages[0]['content'] is string -%}\n at row 1, column 1:\n{{ bos_token }}\n^\n{%- if messages[0]['role'] == 'system' -%}\n","type":"server_error"}

Scout

got exception: {"code":500,"message":"Trying to access property '0' on null! at row 18, column 33:\n    {%- else %}\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n                                ^\n    {%- endif %}\n at row 18, column 33:\n    {%- else %}\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n                                ^\n    {%- endif %}\n at row 18, column 73:\n    {%- else %}\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n                                                                        ^\n    {%- endif %}\n at row 18, column 9:\n    {%- else %}\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n        ^\n    {%- endif %}\n at row 17, column 16:\n        {%- set system_message = messages[0]['content']|trim %}\n    {%- else %}\n               ^\n        {%- set system_message = messages[0]['content'][0]['text']|trim %}\n at row 15, column 5:\n    {%- set user_provided_system_message = true %}\n    {%- if messages[0]['content'] is string %}\n    ^\n        {%- set system_message = messages[0]['content']|trim %}\n at row 13, column 42:\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n                                         ^\n    {%- set user_provided_system_message = true %}\n at row 13, column 1:\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n^\n    {%- set user_provided_system_message = true %}\n at row 1, column 1:\n{{- bos_token }}\n^\n{%- if custom_tools is defined and custom_tools%}\n","type":"server_error"}

[ochafik]

@remon-nashid ooh interesting pattern, will try and accommodate that in minja (looks like those templates expect the system message’s content to be a list, oh well)

[remon-nashid]

@ochafik please scratch that. I'm not sure if this is a regression nor if it's related to the original issue. The errors occurred when I passed an empty string as system message via OpenAI completion API. But if it helps, the issue occurred with Scout and Gemma, but not with other models eg Qwen3-30B or Glm4.5-110b.

[ajunca]

Don't know if it is of any use, but happening with other models too (ex. GLM 4.5 Air):

got exception: {"code":500,"message":"Value is not callable: null at row 56, column 70:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n                                                                     ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 56, column 72:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n                                                                       ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 56, column 85:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n                                                                                    ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 56, column 106:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n                                                                                                         ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 56, column 108:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n                                                                                                           ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 56, column 9:\n    {%- if '</think>' in content %}\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n        ^\n        {%- set content = (content.split('</think>')|last).lstrip('\\n') %}\n at row 55, column 36:\n{%- else %}\n    {%- if '</think>' in content %}\n                                   ^\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n at row 55, column 5:\n{%- else %}\n    {%- if '</think>' in content %}\n    ^\n        {%- set reasoning_content = ((content.split('</think>')|first).rstrip('\\n').split('<think>')|last).lstrip('\\n') %}\n at row 54, column 12:\n    {%- set reasoning_content = m.reasoning_content %}\n{%- else %}\n           ^\n    {%- if '</think>' in content %}\n at row 52, column 1:\n{%- set content = visible_text(m.content) %}\n{%- if m.reasoning_content is string %}\n^\n    {%- set reasoning_content = m.reasoning_content %}\n at row 48, column 35:\n{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not content.endswith(\"/nothink\")) else '' -}}\n{%- elif m.role == 'assistant' -%}\n                                  ^\n<|assistant|>\n at row 45, column 1:\n{% for m in messages %}\n{%- if m.role == 'user' -%}<|user|>\n^\n{% set content = visible_text(m.content) %}{{ content }}\n at row 44, column 24:\n{%- endfor %}\n{% for m in messages %}\n                       ^\n{%- if m.role == 'user' -%}<|user|>\n at row 44, column 1:\n{%- endfor %}\n{% for m in messages %}\n^\n{%- if m.role == 'user' -%}<|user|>\n at row 1, column 1:\n[gMASK]<sop>\n^\n{%- if tools -%}\n","type":"server_error"}