MakeSomethingNew Launches Internal LLM Infrastructure Construction Service

MakeSomethingNew Launches Internal LLM Infrastructure Construction Service

MakeSomethingNew has launched a service to build internal LLM infrastructure, supporting large enterprises in effectively leveraging generative AI and AI Agents.

This service centralizes LLM usage from internal applications, chat UIs, and AI Agents, establishing a common infrastructure for authentication and authorization, model routing, usage log retrieval, and audit compliance. The architecture is designed to allow flexible usage of multiple LLM providers, including Azure OpenAI, OpenAI API, Groq, and OSS LLMs, based on specific needs and situations.

Generative AI adoption in enterprises is shifting from individual or departmental trial usage to full-scale operations connected to business systems, internal knowledge bases, and workflows. However, this has created challenges for company-wide deployment, including user access management, model-specific cost and latency management, audit log acquisition, and securing failover routes during outages.

Leveraging its expertise in generative AI, AI Agent construction, and web development, MakeSomethingNew assists in the design and construction of common infrastructure that allows companies to safely integrate LLMs into their business operations.

### Background

While many companies are adopting generative AI such as ChatGPT, implementing disparate AI tools or API integrations across departments has led to several issues:

- Difficulty in tracking who is using which model for specific tasks - Fragmented authentication, logging, and security designs across departments - Vulnerability to LLM API rate limits or service outages affecting business operations - Inability to cross-manage costs, token usage, latency, and errors - Challenges in safely connecting AI Agents to internal APIs and business data

To address these issues, this service constructs a common LLM infrastructure for the entire enterprise rather than relying on individual implementation for every application.

### Key Features

1. Authentication and Authorization Integrated with Internal IDs

By integrating with internal ID systems like Entra ID, the service controls available models and usage scenarios based on users, departments, and application roles. This prevents unmanaged personal usage while ensuring generative AI adoption adheres to corporate compliance.

2. Routing to Multiple LLM Providers

It bundles multiple providers such as Azure OpenAI, OpenAI API, and Groq, distributing requests based on model characteristics, rate limits, response speeds, costs, and service health. It supports switching to failover destinations if a specific model or region experiences an outage.

3. Visualization of Usage and Audit Logs

It enables tracking of token usage, latency, errors, and costs by user, department, application, and model. This allows operations, security, and business teams to refer to the same logs to monitor usage, improve performance, and handle audits.

4. Internal Connectivity Infrastructure for AI Agent Construction

Beyond simply relaying LLM APIs, the service provides manageable connectivity for internal data, tools, and business APIs referenced by AI Agents. This allows development teams to focus on agent design specialized for specific business tasks rather than common functions like authentication and logging.

### Intended Use Cases

This service is intended for companies and departments that:

- Want to promote generative AI adoption across the organization while addressing security and audit concerns - Wish to deploy AI Agents across multiple departments - Want to flexibly use multiple LLMs (Azure OpenAI, OpenAI, Groq, etc.) based on use cases - Aim to integrate LLMs into internal chat, business systems, and knowledge bases - Need to centrally visualize LLM usage, costs, latency, and errors

### About MakeSomethingNew

MakeSomethingNew is a company dedicated to 'creating something new,' providing system development for NPOs, generative AI support, and web development. Through development services utilizing generative AI and LLMs, AI Agent construction, prompt engineering, and AI integration into business systems, the company supports new business development and operational improvements for organizations.