OPAIRS SQL Agent: Comparing Six LoRA Adapters for Industrial Databases

An industrial data landscape is rarely a single system. PostgreSQL databases for maintenance and production, T-SQL/MSSQL servers on Windows Server for MES and ERP, Apache Iceberg and Spark data lakes for historical production and energy data – and on top of that, often an SAP system with its own table model. A generic language model does not really know any of these dialects well, and it certainly does not know the customer-specific tables, extensions and naming conventions that have emerged in every grown manufacturing IT environment.

This is where the real problem of industrial text-to-SQL applications begins: it is not enough to produce syntactically correct SQL. A model has to understand which tables and fields actually belong together, which SQL dialect it is currently working with – PostgreSQL, T-SQL or Spark/Iceberg SQL – and how the business relationships between production, maintenance, quality, logistics and engineering are represented.

To this end, OPAIRS tested six domain-specifically fine-tuned LoRA adapters against each other. The goal was not to find the largest model. The search was for the candidate that can run reliably as a specialized SQL agent on a single GPU at the customer site across PostgreSQL, T-SQL/Windows Server and Apache Iceberg, and can then be integrated into a broader industrial data landscape.

Six adapters, the same dataset, the same benchmark

All six adapters are based on the same hardened training dataset of 10,939 question-answer pairs. What differs is the respective base model: Qwen2.5-Coder-7B-Instruct, Qwen3-8B, XiYanSQL-QwenCoder-7B, Mistral-7B-Instruct-v0.3, as well as IBM Granite-4.1-8B and Granite-4.0-H-Tiny.

All candidates were evaluated against the same 26-question catalog. The focus is on the database systems that actually run side by side in production industrial environments: PostgreSQL (CMMS/maintenance, production data), T-SQL on Windows Server (MES, ERP) and Apache Iceberg/IOMETE data lakes (production and energy data). In addition, the catalog covers SAP data models (PM, PP, QM, MM) as well as ABAP – an additional strength of the model, but not the core of the comparison. Each answer was scored manually on a scale from 0 to 10; an answer counts as passed at a score of 7 or higher.

The comparison therefore does not measure which base model performs best on a general public SQL benchmark, but which model most reliably solves the industrial database tasks relevant to OPAIRS – across dialects and systems.

The current catalog covers only a portion of the future target environment. Production industrial data landscapes rarely consist of a single database; they combine PostgreSQL, T-SQL/Windows Server, Apache Iceberg data lakes, ERP, MES, PLM and CMMS systems and other proprietary specialist systems. It is precisely this breadth that is to be covered step by step in the next validation stages.

Two Granite adapters lead the field

The result is closer than the different model architectures would suggest:

• OPAIRS-Granite-8B-Instruct and OPAIRS-Granite-Tiny-Instruct: each 7.87/10 and a 76.9% pass rate, corresponding to 20 of 26 questions passed. The identical hit rate despite very different architectures is remarkable: a dense 8B model on one side, a hybrid MoE with roughly 1 billion active parameters on the other.
• OPAIRS-Coder-7B-Instruct: 7.79/10, also at a 76.9% pass rate. This adapter shows the most balanced SQL quality across PostgreSQL, T-SQL and Spark/Iceberg SQL.
• OPAIRS-Qwen3-8B-Instruct: 7.60/10 at a 73.1% pass rate.
• OPAIRS-XiYan-7B-Instruct: 7.25/10 at a 65.4% pass rate. At the same time, XiYan is the fastest adapter in the original comparison.
• OPAIRS-Mistral-7B-Instruct: 2.06/10 and not a single question passed. Self-correction spirals, token-level corruption and 16 answers that break off mid-sentence rule this adapter out for production use in its current configuration.

The most important finding, then, is not that a single model wins by a wide margin. Three adapters are relatively close in quality. The actual selection is therefore decided only by the interplay of answer quality, inference time and available memory on the future target hardware.

Ranking: two adapters tied for first place

Despite fundamentally different architectures, Granite-8B-Instruct and Granite-Tiny-Instruct achieve exactly the same score and pass rate across PostgreSQL, T-SQL and Iceberg questions. Coder-7B-Instruct follows closely behind with the most balanced dialect coverage. XiYan-7B-Instruct is significantly faster but weaker in hit rate. Mistral-7B-Instruct is the only candidate to be set aside entirely. For comparison, the chart also includes Claude Opus 4.8, GPT-OSS-20B and both Granite base models without an adapter.

Fine-tuning beats the local untuned model – but not every external reference

For context, two models without domain-specific SQL training were also tested against the same catalog. GPT-OSS-20B, run locally and untuned, achieves a pass rate of 65.4%, while the best in-house adapters reach 76.9% – fine-tuning thus delivers a measurable advantage of just over 11 percentage points.

Against Claude Opus 4.8, the picture changes: the hosted flagship model reaches an 80.8% pass rate and an average score of 8.08/10, putting it ahead of every in-house adapter in this benchmark – without any SQL-specific fine-tuning. For the architecture decision, though, this comparison is only a reference point: a hosted model with per-token API costs, external processing and no offline operation fills a different role than a specialized adapter that runs entirely locally at the customer site.

A different result of the comparison is therefore more interesting: on two questions about SAP-specific table joins, practically all eight tested models deliver their weakest answers – including Claude Opus 4.8. When several independent model families fail at the same point, the cause does not necessarily lie in the model. In this case, the result points to gaps in the underlying schema and context knowledge – and these gaps will be addressed specifically before the next evaluation round.

The biggest lever was not the model, but the sampling temperature

After the first comparison, the two Granite finalists were additionally examined on the actual vLLM serving stack. This revealed a clear effect: a large share of the quality differences initially observed stemmed not from the model but from the inference configuration.

At the original temperature of 0.2, the pass rate was only around 61.5%. Pure greedy sampling with temp=0.0 already raises it to 76.9%. A slightly adjusted setting with temp=0.1, top_p=0.95 and a repetition penalty of 1.05 reaches the same pass rate with average scores between 7.60 and 7.84/10. FP8 versus bf16, by contrast, practically does not change the result.

The effect becomes even more pronounced at higher temperatures: from temp=0.7 onward, the pass rate drops by more than 35 percentage points for both architectures. For production use, it is therefore decisive not only which model is deployed, but equally which sampling configuration it is run with.

The comparison with the respective base models without an adapter also confirms the effect of fine-tuning: both untuned base models reach only a 53.8% pass rate. The largest gains come in particular with domain-specific questions and customer-specific data structures.

Target hardware instead of a theoretical model comparison

A benchmark alone does not decide which model is deployed at the customer site. The target hardware for the SQL agent is deliberately a single RTX PRO 2000 Blackwell with 16 GB of VRAM. The card needs no additional PCIe power connector, occupies only one slot and can run natively under Windows. The goal is therefore no additional data center infrastructure – the specialist agent should be able to run in a standard workstation or a compact server directly at the customer site.

Target hardware: single-card workstation instead of a data center

On a workstation with a single RTX PRO 2000 Blackwell (16 GB VRAM), the two Granite finalists were compared again. OPAIRS-Granite-8B-Instruct achieves 7.84/10 at an average response time of about 6.2 seconds, OPAIRS-Granite-Tiny-Instruct 7.60/10 at about 1.5 seconds – roughly four times faster at the same 76.9% pass rate. The measured quality difference lies within the run-to-run variance and is therefore currently not a sufficiently robust reason to accept the significantly longer inference time of the larger model.

Granite-Tiny is the choice for the single-card rollout

For the planned single-card rollout, Granite-Tiny is therefore preferred. The reason is not only the roughly four times shorter response time: the smaller model also leaves more memory free for the KV cache. On the same 16 GB GPU, more parallel requests can thus be processed before additional hardware is required – and in production, this headroom matters more than a small difference in average score.

Granite-8B-Instruct nonetheless remains part of the stack: if a specific use case clearly prioritizes maximum answer quality over throughput and concurrency, the larger model remains a sensible alternative. The decision is therefore not "the small model is better than the large model", but rather: for the intended hardware and the expected load profile, Granite-Tiny currently delivers the better overall package.

The next quality step is hardening, not automatically a larger model

At the same time, the comparison shows very concretely where the next development work lies. Some of the errors stem from missing context about tables, relationships and business data models. Other errors can be detected automatically and are therefore suitable as a reward signal for further training steps – for example, aggregations over already pre-filtered individual rows, or filter conditions that are computed inside a CTE but not applied in the main query.

In addition, there are errors that occur even with external reference models. They do not explain the gap between the models, but they remain distinct hardening targets. And finally, there are pure deployment insights such as the high sensitivity to sampling temperature. None of these points automatically calls for a larger base model – the next quality leap will come from targeted hardening of the dataset, better context information and a controlled inference configuration.

RAG was not yet part of the evaluation in the benchmark so far

An important point when interpreting the results so far: Retrieval-Augmented Generation, or RAG for short, was deliberately not yet taken into account in this benchmark. The values so far therefore measure exclusively the quality of the fine-tuned models and their current inference configuration.

This matters because a production industrial agent should in future not rely solely on the knowledge it learned during fine-tuning. When needed, it should receive additional context from a company's own systems and knowledge sources – for example data models and documentation from PostgreSQL, MES, PLM or CMMS systems, as well as table descriptions, technical documentation, process knowledge, database metadata or other approved company information.

In industrial IT landscapes in particular, this context is decisive: two companies can use the same database or MES software and still have completely different table structures, extensions, naming conventions and process logic. Training all of this knowledge into a base model is neither realistic nor sensible. This is exactly where RAG is meant to add a further layer: for each request, the agent receives precisely the information needed for the specific task. The goal is not to hand as much context as possible to the model, but to provide the right context at the right time.

RAG will be tested with its own validation catalog

With RAG, too, we believe it is not enough to be able to answer a handful of demo questions successfully. OPAIRS is therefore currently developing a separate validation catalog for the retrieval layer. It is intended to systematically examine: Which additional pieces of information actually improve an answer? Which documents or metadata are found reliably? When does additional context lead to a better SQL answer – regardless of dialect – and when does retrieval merely produce more context without improving the business quality?

Fine-tuning and RAG working together

The future target environment comprises not just a single system, but a variety of industrial databases and sources: PostgreSQL, T-SQL/Windows Server, Apache Iceberg data lakes, ERP, MES, PLM, CMMS and technical documentation. The long-term goal is an architecture in which fine-tuning and RAG take on different tasks: fine-tuning gives the specialist agent patterns, dialect mastery and a fundamental understanding of the domain, while RAG supplies the concrete, current and customer-specific context for each request – orchestrated in the same multi-GPU pipeline setup in which the other OPAIRS specialist agents run.

26 questions are enough for selection – not for production release

The current benchmark provides a good first comparison between the adapters. For a robust production release, however, a catalog of 26 questions is too small. The next step is therefore a validation catalog of more than 1,000 questions, intended to cover the actual breadth of production industrial data landscapes: different PostgreSQL, T-SQL/Windows Server and Apache Iceberg environments, ERP, MES and PLM structures, maintenance and quality information, customer-specific table models and more complex query chains.

Only at this scale can we reliably identify which errors occur systematically and where a further fine-tuning round, better retrieval or additional context is needed. At the same time, this creates the basis for evaluating fine-tuning and RAG together rather than in isolation. The decisive question in future is not only whether the model can solve the task, but whether the overall system can find the right information, classify it correctly and produce a technically sound answer from it.

This requires real industrial questions

At exactly this point, internal development alone is not enough. A generically built test catalog can expose many technical errors, but it cannot fully capture how differently real manufacturing companies have set up their PostgreSQL, MES, PLM, CMMS and ERP systems. OPAIRS is therefore looking for partners and customers interested in validating this next development stage together.

We are not looking for perfectly prepared demo datasets. Real questions from existing system landscapes are especially valuable: What information does a production manager actually need? How does a maintenance technician ask about historical faults? How are production orders, materials, quality data and machine states linked? Which relationships between the various databases and systems are decisive for an answer? And which answers are plausible from a technical standpoint but still wrong or incomplete in operational reality? This feedback is crucial for further development.

High-quality feedback becomes part of the optimization

The goal of the collaboration is not to collect as many test questions as possible – the quality of the feedback is what matters. For every relevant answer, it should become clear: What was correct? What was wrong from a business perspective? What context was missing? What information should the system have found? Was the problem in the model, in the retrieval, in the available metadata or in the data source itself?

This feedback can then flow in a structured way into the next development cycles. It yields new evaluation cases, improvements to the retrieval logic, additional training examples and targeted hardening measures. In this way, a general specialist agent gradually evolves into a system that understands real industrial data landscapes ever more reliably.

Two paths for joint validation

Validation can take place directly on a local OPAIRS installation at the partner site. Alternatively, tests can be run on the local OPAIRS development infrastructure against a controlled replica or an appropriately prepared data structure. In both cases, data processing remains local or under the control of the respective partner. The results developed jointly feed directly into the further development of the evaluation catalog, the fine-tuning datasets and the RAG architecture.

For OPAIRS, this is a central part of the next development phase: an industrial agent does not get better simply by enlarging the base model. It gets better when model, context, data structure and business feedback are optimized together.

Production and maintenance agents come next

The SQL agent is the first of the specialized agents that complement GPT-OSS-20B within the OPAIRS architecture. The principle remains the same: the main model handles general interaction and orchestration, while specialized tasks are handed off to smaller models fine-tuned for their respective domains.

Using the same methodology as for the SQL agent, evaluations for the production and maintenance agents are currently underway: domain-specific LoRA fine-tuning, scoring against expert catalogs and comparison with untuned reference models. In parallel, we are investigating how these specialist agents can in future obtain additional context via RAG from ERP, MES, PLM, CMMS and other industrial information sources.

The results will be published in upcoming Insights. What will matter is not only the quality of the individual specialist models, but their interplay within the existing multi-GPU orchestration with GPT-OSS-20B, the retrieval layer and the respective company data.

For technical collaboration, joint validation or industrial projects: office@opairs-systems.com