#llm #machinelearning #artificialintelligence
A look at the current state of running large language models at home. Running 13B and 30B models on a PC with a 12gb NVIDIA RTX 3060.

KoboldCPP:
[ Ссылка ]

KoboldAI:
[ Ссылка ]

Oobabooga UI:
[ Ссылка ]

Link to TheBloke's Huggingface page:
[ Ссылка ]

Reinstall llama-cpp-python with CUBLAS/CUDA support:
pip uninstall -y llama-cpp-python
CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir

свернуть