- JoBS, presented as an oral at an ICML 2026 workshop, cuts LLM fine-tuning costs by training a scaling-law-inspired performance predictor that estimates full training outcomes from a handful of steps, replacing the need for a complete training run at every optimization iteration.
- In a wet-lab battery electrolyte task, the LGBO framework reached 90% of best observed performance in 6 iterations versus more than 10 for standard Bayesian Optimization, a roughly 40% reduction in iterations that holds across both synthetic benchmarks and physical experiments.
Four new frameworks presented at ICML 2026 and ICLR 2026 are converging on the same idea: stop running expensive LLM training jobs to evaluate each candidate configuration, and use the LLM itself to steer the search. The most concrete result so far is JoBS, which co-optimises data and model configurations using a cheap performance predictor rather than full training runs, and outperforms both independent optimisation methods and existing multi-fidelity Bayesian Optimisation baselines across a range of downstream tasks.
JoBS Rethinks LLM Training Cost
Fine-tuning an LLM for a specific downstream task means finding the right combination of data mix and model configuration, a search that traditionally requires a full training run for every candidate configuration evaluated. Bayesian Optimisation handles this systematically, using observed performance to steer toward better settings, but the cost per evaluation has historically made thorough exploration of the joint data-model space impractical.
The paper introducing JoBS, presented as an oral at ICML 2026’s “Decision-Making from Offline Datasets to Online Adaptation” workshop and co-authored by Bryan Kian Hsiang Low and colleagues, sidesteps that cost with a two-phase approach. First, a fraction of the optimisation budget trains a scaling-law-inspired performance predictor, one that estimates what a fully trained model would score after seeing only a small number of training steps. Second, Bayesian Optimisation runs against that predictor rather than against full training jobs. The net effect is that far more configurations get evaluated within the same compute envelope.
Empirically, JoBS outperforms both methods that optimise data and model settings independently and existing multi-fidelity BO baselines across a diverse set of downstream tasks. That combination, lower cost per evaluation and broader coverage of the search space, makes joint optimisation viable for teams that previously had to choose between thorough search and staying within budget. For a closer look at how LoRA-based fine-tuning sits alongside these approaches on the cost curve, the trade-offs are worth weighing.
LLMs Supercharge Bayesian Search
JoBS sits within a broader wave of work that treats LLMs as active participants in the optimisation loop rather than passive targets of it. A paper posted to arXiv on October 2, 2026, introduces two related frameworks, AuxBO-Evolve and AuxBO-Vicinity, which feed LLMs the kind of information classical BO discards: full training curves, diagnostic logs, expert annotations and prior literature. Classical BO has no mechanism for ingesting unstructured text; AuxBO uses the LLM’s semantic reasoning to evolve beliefs about optimal regions or to locally guide the acquisition function. The paper reports consistent gains over standard BO and existing LLM-augmented baselines across tasks ranging from synthetic benchmarks to nuclear fusion experiments.
The LGBO framework, presented at ICLR 2026 in April and posted to arXiv in May, takes a different approach. Rather than using LLMs for warm-start or candidate generation, LGBO embeds LLM-driven preferences into every iteration through what the authors call a “region-lifted preference mechanism,” which shifts the surrogate model’s mean in a stable, controllable way. In a wet-lab task optimising Fe-Cr battery electrolytes, LGBO reached 90% of the best observed value in 6 iterations; standard BO and other LLM-augmented baselines needed more than 10. That is roughly a 40% cut in iterations for comparable performance.
LABO, another ICML 2026 framework, approaches the problem from the cost side. It treats the LLM as a cheap virtual screening layer: broad exploration first, using inexpensive LLM predictions to identify promising regions, then selective deployment of costly physical experiments only where the model’s predictions are uncertain. A probabilistic disagreement measure between LLM predictions and experimental outcomes determines where real experiments are worth running, which keeps expensive evaluations concentrated in genuinely informative regions.
What This Means for Practitioners
Taken together, JoBS, AuxBO, LGBO and LABO point in a consistent direction: the compute-heavy brute-force approach to configuration search has a cheaper alternative, and that alternative is now producing replicable results across multiple venues. Parameter-efficient fine-tuning methods like LoRA and QLoRA already shifted the cost curve for model adaptation; these BO frameworks extend that logic to the search process itself.
The practical implication is that teams constrained by compute budget can now run meaningfully broader searches over data mix and model configuration without proportionally larger training bills. Infrastructure like Kubeflow’s Katib already handles automated hyperparameter optimisation in Kubernetes environments; plugging a predictor-based BO framework into that infrastructure is a shorter step than rebuilding search from scratch. The open question is how well these predictors generalise across model families and task types, something none of the papers currently quantifies at scale.
