Harvey vs Legora | When Are Terrible Gross Margins OK?

Plus: Where AI costs belong on the P&L and what actually goes into COGS

Brought to you by…Remote

Global hiring gets complicated fast when Finance can’t clearly answer two questions: What will this cost us, and who owns the compliance risk?

• With Remote, pricing is transparent and predictable, with no hidden fees.

• With Remote, uncapped indemnification gives stronger protection against compliance risk.

Fewer surprises. Clearer ownership. Better visibility into global employment spend and risk.

---

The Harvey vs Legora Battle

I love a good public rivalry with juicy social media trolling. The Harvey-Legora battle is the Kendrick-Drake feud for those who know what “Pari Passu” means.

Harvey and Legora are both only a few years old and have seen epic revenue growth, which has led to skyrocketing valuations and large consecutive fundraises.

Both companies were among the very fastest application layer AI companies to reach $100M (and $200M) ARR. Legora went from $1M → $100M ARR in 18 months and then $100M → $200M ARR in 6 months…

Both have raised tons of money to fuel rapid growth, and both are burning cash to sustain that growth (more on this below).

But these two companies have many doubters about their long-term future. The comments are valid concerns:

1. They are just thin AI wrappers

2. They have negative/low gross margins so they will never be profitable

3. The products are worse than just using Claude/ChatGPT

4. How will they actually differentiate? The frontier models are entering their space

Is Eric (founder of one of the fastest growing law firms) right?

Do Gross Margins Matter?

Let’s start with the gross margin point because it is a key financial metric underlying the broader discussion of these companies.

AI products have significantly lower gross margins than their legacy SaaS friends, but they appear to be improving!

ICONIQ

An issue though is how much gross margins can differ between AI products/companies. The range is so much wider than with legacy SaaS. Many even have negative gross margins.

Harvey and Legora have both been called out for weak gross margins:

Harvey’s Terrible Gross Margins

Harvey’s gross margins reportedly fell to -50% in June because:

1. Harvey primarily has flat seat-based pricing (doesn’t change with AI usage)

2. Underlying frontier models have been getting exponentially better, which drives more usage since it’s more useful (especially since all additional usage is free to customers)

3. Harvey released better AI capabilities earlier this year that drove more usage

Keep reading to see how Harvey flipped to positive gross margins within one quarter (50+ percentage-point increase 🤯)

Legora “Positive” Gross Margins

The CFO of Legora said the following about their gross margins in a post about the most important metrics for AI companies.

Having “positive gross margins” is better than negative but also a pretty low bar. A company with 0% long-term gross margins is worth $0.

I assume the gross margins are pretty low because their CFO specifically quantified other metrics (300% NRR and 95% GRR) while just saying gross margins are positive. Barely positive gross margins are also not sustainable given all the OpEx spend these AI companies have. So improvements still must come.

The second part of this Tweet reads like a clear jab at Harvey’s period of negative gross margins :) - “There’s a shorter route: Price below what it costs to serve”.

A Critic’s View of Negative Gross Margins

The skeptics will say:

Of course you can have explosive revenue growth if you are selling $1s for $0.50

There is unlimited demand for compute and frontier intelligence so if someone is willing to give you that same intelligence for a 50% discount then you will have almost unlimited demand. No brainer, right?

But what happens when gross margins improve and approach 0%? Probably no change. If there is unlimited demand for the underlying intelligence that you are reselling, then there will be near unlimited buyers for your thin AI wrapper at any discount.

But a truly thin AI wrapper will lose almost all demand once gross margins start to become positive because users will just switch to the underlying AI models since using the AI models directly would presumably be cheaper at that point.

A Positive View of Negative Gross Margins

What if temporarily negative (or very low) gross margins can create a longer-term strategic advantage that enables high gross margins?

This is obviously the bet of high-growth negative gross margin companies.

Here are a few arguments:

1. S&M driven - Drive initial customer adoption into a free/cheap tier to eventually get high overall gross margins by pushing the biggest customers into a profitable tier.

2. Build Adoption/Distribution - Get the adoption, get more data, and continue to add value on top of what the AI models can provide. Vertical software (like Legal AI) is a great place to do that.

3. Product research - Give away frontier-level intelligence so the product is great while you figure out how the product is used to make improvements while bringing down costs (fine-tune your own model, improve harness, or model routing, etc)

4. Falling Inference Costs - Inference costs are falling fast enough to underwrite current gross margin losses (costs have come down 10-100x in the last couple of years)

#4 is a bad reason to explain bad current gross margins. Yes, inference costs are falling but the AI model vs Legal AI (or a “thin AI wrapper”) decision is relative to the AI model costs… If AI model costs come down 100x then customers are going to just use the AI models directly if you remain a thin AI wrapper.

Improving AI Gross Margins

Companies have three options to improve gross margins based on their current product offering:

1. Change the pricing model

2. Solve the LLM cost issue

3. Cap AI usage so costs don’t exceed fixed pricing

Longer-term, you really need gross margins improving from your product differentiation and the material value you are adding beyond the LLMs.

Let’s look at how Harvey and Legora are attacking their gross margin problem:

Harvey’s Gross Margin Fix

Harvey’s gross margins clearly needed a major fix after it went deeply negative.

Harvey is shifting more to a consumption-based model, but didn’t want to force customers on it too soon so they focused first on #2 (solving the LLM cost issue)

Harvey made several changes to reduce AI costs, including releasing their own model in August that was fine-tuned from an open-weight model (Kimi K3, from the Chinese lab Moonshot).

• Claims their post-trained model can deliver better answers than the frontier in many cases

• Still uses frontier models when needed

• Their post-trained AI model costs can be up to 1/10th the cost of using just the frontier models:

  • Kimi K3 is 40% cheaper

  • Harness improvements

  • Orchestration improvements

If Harvey can offer their solution at a better price than frontier AI models (with good gross margins) and with equal or better responses for legal teams then of course every legal team will prefer their solution.

Legora’s Gross Margin Improvements

Legora’s “positive gross margins” are better but also not sustainable if they are really low, which they presumably are. They still have to pay for all the costs in R&D, S&M, and G&A….

Legora chose a combination of #1 and #2 (consumption pricing and solving LLM costs)

Legora introduced a consumption-based pricing model a couple of months ago for certain customer segments, so pricing will move more in line with customer usage.

Legora isn’t currently building their own model like Harvey. They are focused on some post-training for specific tasks, improving their harness, better orchestration, and model routing.

The CTO of Legora wrote a post about how “The best model is all the models” - clearly trying to differentiate themselves. What he said about the vertical AI opportunity makes a lot of sense:

Vertical AI companies are uniquely positioned to map the nuanced model map and spot “model arbitrage” opportunities - places where changing models gives more performance, lower cost, lower latency or the gain of other desired properties - that aren’t immediately clear from a public leaderboard. They can do this because they have two very valuable things: in-house expertise and valuable real-world usage.

What AI Stuff Goes Into Gross Margins?

Gross margins represent the ceiling on profitability.

Gross margins show how profitable a company can be if there were no other expenses (stop developing products, stop fine-tuning AI models, no sales reps, AI automatically does finance, etc.).

Below is a list of the main expense categories that go into COGS (and therefore impact gross margins). I also broke out other AI-specific expenses that should be in other sections of the P&L to show what specifically shouldn’t be in COGS.

I know a lot of companies that are getting the P&L classification of these AI costs wrong. Some common ones I have seen are:

• Fail to allocate AI model subscriptions (and dump it all to G&A)

• Free trials aren’t separated so it all goes into COGS. Note - not everything given away for free is S&M. Talk to your accountants.

• Accounting team doesn’t understand what vector databases are (or other data processing, retrieval or agent execution tools) so it all goes to R&D

• Some R&D costs may need to be capitalized under GAAP and then they get amortized into COGS. Yes, accounting can be confusing…

Final Thoughts

I will leave you with three things to think about:

1. Many generational AI companies are being built, but I agree with Harry that this feels like 2021 on steroids. Many financial metrics are being ignored in the pursuit of revenue growth. The thing is…the good companies aren’t ignoring them. They are very much paying attention to them, but they are temporarily sacrificing some metrics to build a long-term advantage. But many simply have terrible unit economics.

2. Gross margin reporting is important. If you aren’t properly tracking gross margins, your company will fail to make the right long-term decisions. Granular reporting is important - down to the product and even feature level

3. Thin AI Wrapper Test: What is the correlation between rising gross margins and sales demand? If demand falls off a cliff as gross margins go from 0% → 40% then it’s probably still a thin AI wrapper.

The question is whether today’s losses are building tomorrow’s advantage…or if it’s a continuous VC fundraising treadmill that likely ends in valuation destruction and tears.

---

Footnotes:

• Remote can help you hire globally while remaining compliant and providing transparent pricing (today’s sponsor)

• Subscribe and share with your friends

*Disclaimer: Informational and educational purposes only.

Bonus:

Saw the below comparison between Harvey and Legora so sharing if helpful. Not sure how the survey was done (and sample size is pretty small), but still interesting. Obviously leans toward Harvey but Legora is very close on all of these metrics.