server: docs: --threads and --threads, --ubatch-size, --log-disable
#6254
Conversation
| - `-ts SPLIT, --tensor-split SPLIT`: When using multiple GPUs this option controls how large tensors should be split across all GPUs. `SPLIT` is a comma-separated list of non-negative values that assigns the proportion of data that each GPU should get in order. For example, "3,2" will assign 60% of the data to GPU 0 and 40% to GPU 1. By default the data is split in proportion to VRAM but this may not be optimal for performance. Requires cuBLAS. | ||
| - `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `512`. | ||
| - `-b N`, `--batch-size N`: Set the batch size for prompt processing. Default: `2048`. | ||
| - `-ub N`, `--ubatch-size N`: physical maximum batch size. Default: `512`. |
There was a problem hiding this comment.
Should we also make it clear that ubatch should be enough when using for embeddings?
There was a problem hiding this comment.
Yes, there is no advantage to increasing n_batch above n_ubatch with embeddings models with pooling, because the entire batch must fit in a physical batch (ie. n_ubatch). n_batch is always >= n_ubatch.
There was a problem hiding this comment.
@ngxson I think it is better to add this check this in server.cpp. i will create an issue to trace it, we will implement it later on.
There was a problem hiding this comment.
The embeddings from multiple slots can go in a single batch. For example with n_batch = 2048 and n_ubatch = 512 we can process 4 full slots in one go
There was a problem hiding this comment.
It could be implemented, but the batch splitting code does not take this into account. llama_decode will just fail if n_tokens > n_ubatch.
There was a problem hiding this comment.
We can move this discussion in ?
There was a problem hiding this comment.
Ah yes sorry, ignore my comment
Update
server/README.mdwith actual params supported.