Download
Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop. Xorbits Inference(Xinference) is a powerful and versatile library designed to serve language, speech recognition, and multimodal models. With Xorbits Inference, you can effortlessly deploy and serve your or state-of-the-art built-in models using just a single command. Whether you are a researcher, developer, or data scientist, Xorbits Inference empowers you to unleash the full potential of cutting-edge AI models.
Features
- Simplify the process of serving large language, speech recognition, and multimodal models
- You can set up and deploy your models for experimentation and production with a single command
- Experiment with cutting-edge built-in models using a single command
- Make the most of your hardware resources with ggml
- Offer multiple interfaces for interacting with your models, supporting RPC, RESTful API(compatible with OpenAI API), CLI and WebUI for seamless management and monitoring
- Excel in distributed deployment scenarios, allowing the seamless distribution of model inference across multiple devices or machines
Project Samples
