Can models learn hidden pricing rules in a market economy?

I have always loved baseball, and one of the aspects that intrigues me the most is how players are valued, especially in trades. Just this past year my beloved Phillies traded away a hot pitching prospect flying through the minors, Ramon Marquez, in exchange for a rental of Luis Arraez.

This was a deadline trade. Usually, trading a controllable young pitcher who looks destined to be a high leverage reliever in the coming years for a few months of a contact hitter wouldn’t be thinkable. But for the Phillies, who saw this year as an opportunity to make a run in the postseason, their valuation of assets changed. Having Arraez now became more important than Marquez years down the line.

Now, in the Phillies’ case, the trade didn’t work out. Arraez ultimately wasn’t on the roster due to injury for the club’s 3 game loss in the NL Wild Card round. But the trade itself wasn’t illogical to make. GMs make similar moves all the time. There is a lot more involved in the decision to make a move than just a player’s value on paper.

As I’ve been experimenting with LLMs, I wanted to understand how well models understand these hidden pricing rules when they’re in the shoes of a GM, and whether or not we can teach a model how these rules work with PEFT.

Experiment design

To get started, I needed to understand how well a model understands these dynamics unmodified. For our experiment, I decided to use Nemotron 3 Nano 4B [1]. I’ve been experimenting with the NeMo toolkit lately, and this model is efficient to train with NeMo Gym [2] for our later RL experiments.

I wanted to design an experiment that would be easy to measure, and ensure the model wouldn’t be able to learn the hidden value of the player from their statline alone.

The design I ultimately decided on was a simulated trade deadline. The model makes one single take-it-or-leave-it offer for a player at the trade deadline, or takes no action.

Here’s an example of what the agent sees for a single turn:

Target: left-handed reliever, 2 extra year(s) of club control.
Public consensus value: 17.1
Your internal value (fit + need): 32.0
Seller: a fringe contender club.
Rival bidders known to be in on this player: 1
Days until the deadline: 9
Make your move.

This card is supposed to represent what a general manager would know going into a trade deadline. The model has a single offer they can make for a player, or they can choose to walk away and take no action. In the market simulation, the selling club has a private asking price, r. If the model’s offer called p is at least r the deal is executed. The model’s club gains V - p where V represents what the player is worth to that club internally.

When presented with the offer, the model can respond with one of two actions:

{ "action": "offer", "amount": X }
{ "action": "walk" }

How is the hidden value in the market quantified?

Every seller in the market has a hidden asking price for a player. This is drawn from a normal distribution, the center of it being the public value of the player noted on the card. The distribution is scaled by a constant for values on the player card according to these terms:

How is it scored?

In this experiment, the grader knows the true value of the player and the seller’s distribution. The reward it calculates given a model’s offer is the expected value of the offer, represented below.¹1 We’re using expected value here because our simulation is a one-shot trade market. Most trades are not likely to happen on their face, so using an outcomes based approach would not provide signal well simply due to the underlying market dynamics. More on the limitations of this approach later.

It will then divide this by the best possible expected value on the player offer which it finds with a 2,000 point grid search.

This means that the value of the reward would be:

• 1.0 for the best possible offer
• 0 for walking away from a deal worth making
• [-1,0) overpaying for a player, with a floor of -1
• -1 if the model outputs a value that cannot be parsed

The below chart plots the seller’s distribution, illustrating where the value lies in the true deadline market.

def reward(task: Task, completion_text: str) -> float:
    valid, offer = parse_action(completion_text)
    if not valid:
        return -1.0
    _, best_ev = optimal_offer(task)  # 2,000-point grid search over (0, V]
    floor = NO_DEAL_EPS * task.public_value  # 1% of the public value
    ev = 0.0 if offer is None else expected_surplus(task, offer)
    if best_ev <= floor:
        # No deal worth making: walking (or a lowball) is correct
        return 1.0 if ev >= 0 else max(-1.0, 1.0 + ev / floor)
    return max(-1.0, min(1.0, ev / best_ev))

How well does the base model perform?

I ran the Nemotron 3 Nano 4B model through two splits, one was a held-out test of players with 0-3 active bidders per player. The other was a shifted market split where 4 or 5 rival bidders were competing for players. Each ran 300 trade scenarios with 8 samples per trade, for a total of 2,400 runs per split.

The way we’ll measure how well the model performs is by calculating how much of the expected value is captured by the model’s offers. This works since we know the hidden value in advance.

For the initial baseline run, I measured how much of the expected value the model captured for the test market (with 0-3 competitors) as well as the shifted market (with 4-5 competitors).

For the test market, the base model captured 21.20% of the potential EV for its scenarios. This fell to 17.62% for the shifted market. The model produced valid actions 100% of the time, but leaves a tremendous amount of economic value on the table. This is a clear indication that the base model doesn’t intrinsically understand the pricing rules in the market in this environment. In other words, there is room to learn the market’s hidden pricing structure.

How can we improve the model’s performance?

One of the first challenges any developer building with AI will experience is a scenario where a model doesn’t reliably output the answer you’d expect when it’s put in a certain environment.

The first line approach for this scenario is to introduce RAG (Retrieval-Augmented Generation) [3]. This is where the application around the model will locate information relevant to the turn that the model is handling, and add that relevant information to the request before it’s sent to the model. It usually uses semantic search over some corpus of information stored in a vector database, but RAG doesn’t have to refer to semantic search, it really can be any kind of context we pull into a request before it’s sent off to the inference provider.

This approach works really well when we have domain-specific data that the app needs to rely on. Imagine a customer support bot that a company’s engineering team might build to increase customer satisfaction. This is a great case for a RAG implementation, because base models aren’t going to have the company’s knowledge base of information in their pre-training data. The bot will use RAG to pull from the company’s knowledge base as needed. This approach also allows developers to have more hands on control over what their model is responding with, since data in an index that RAG relies on can be updated on demand.

In our case, though, RAG isn’t ideal. Why? We want to influence how the model actually behaves. When the experiment is run again, the model still won’t have information about the seller’s hidden price. We want it to understand the dynamics of the market intrinsically.

Enter RL

So, how in the world do we actually change how a model behaves? This is where RL comes into play. For a task like this, a modern technique called Parameter-Efficient Fine-Tuning (PEFT) is a great choice for attempting to subtly influence the behavior of a model without retraining the whole thing.

In our case, I am going to experiment with an RL method called Group Relative Policy Optimization (GRPO) [4] to create a Low Rank Adaptation (LoRA) [5] that modifies only a small subset of the weights of the base model. LoRA is tremendous for experimenting with RL with limited resources. Tools like Unsloth [6] allow training adaptations sometimes on just single consumer GPUs, depending on the base model.

Here though, I decided to use NVIDIA’s NeMo stack for our RL experiment since I entered this project with the goal of getting familiar with NVIDIA’s tooling.

There are two main components to get familiar with when planning on using this stack: NeMo Gym and NeMo RL [7].

How GRPO works in this market

Time to start training. For this GRPO experiment, the model will run against each player card 8 times. Each is scored according to the reward logic above. Runs with higher reward are reinforced.

In other words, for any one given example, we reinforce rollouts where the reward is higher than the average reward for the 8 sample group.

I ran this experiment on Modal. You can see the Modal app entrypoint here in the reference code.

What happened in training

During training the reward jumped quickly and then started to level off. It reached 0.40 over the first 25 steps, and reached 0.92 by step 400. Returns start to diminish around this point, with reward climbing to about 0.95 after step 700.

The below chart illustrates how the reward climbs as the run progresses, through 1,000 steps.

What did the model learn?

So, our reward seems to have risen nicely over 1,000 steps, but how does that actually translate to how the model behaves?

Let’s look at that player card from the original example again.

Target: left-handed reliever, 2 extra year(s) of club control.
Public consensus value: 17.1
Your internal value (fit + need): 32.0
Seller: a fringe contender club.
Rival bidders known to be in on this player: 1
Days until the deadline: 9
Make your move.

If we take a look at what the model offered before the GRPO run, the offers are spread between $17.1M and $32M. Three of them are at the player’s public value of $17.1M, and only one offer (at $24.4M) earned more than 7% of the best surplus.

After training, all eight land between $29.0M and $29.5M, earning 96% or more of the best expected surplus.

Results

If we rerun the whole sample on held out trades, the results show that the model captures 93% of the market’s expected value, versus 21% from before training.

The shifted market

Interestingly, when the model was exposed to the a more competitive market that included 4-5 rival bidders for players, the trained model falls to 78% of expected value recovered.

If we use a simple midpoint rule that averages the model’s value and the public value of the player, it beats the trained model at 87%.

The pitfall here appears to be scenarios where rebuilding clubs are offering a player. The trained model seems to consistently underestimate the willingness of the extra bidders to drive the price up.

Limitations and Next Steps

One of the main limitations of this experiment is that we are using a simulated market. The reward is modeled on expected value here, not outcomes. In a real world market there are additional factors that go into player moves beyond simply the best one-shot offer.

The reward also seemed to plateau here during training.

To take this experiment further, I think my next steps would be to work on redesigning the reward so that it learns the hidden rule about increased bidders leading to higher prices.