comprehensive, scalable ML inference architecture using Amazon EKS, leveraging both Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP

High-performance LLM Gateway built in Go - OpenAI compatible proxy with multi-provider support, adaptive routing, and enterprise features

This repo presents resilience patterns for scaling inference for Generative AI workloads on AWS: Bedrock cross-Region inference, AWS account sharding, and intelligent routing with LLM gateways.

Connect any LLM-powered client app, such as a coding agent, to any supported inference backend/model.

LLMCallGateway 是一个基于 LiteLLM 构建的专业 LLM API 网关服务。将所有模型（包括非 OpenAI 模型）的请求格式统一为 OpenAI 格式，并提供详细的日志跟踪和性能监控，帮助开发者直观了解与下游 LLM API 交互的细节与成本。