I Tested Jev vs Laya on My Inbox (Surprising Results)

Everyone's saying the free Leia beats the paid Jev. Jev is an AI that does one job. You hand it something like an email and it decides what to do with it. Everyone tested it on games like Tetris, so I tested both on the thing that you'd actually use it for, my real inbox. 450 emails sorted by hand, same test for both. Then I trained the free one on my own emails to see if that changes who wins. All right, so here's

How the test works. Every email goes into one of four buckets, urgent, needs a reply, just for my information, or junk. My labels are the right answers. Both models get the same 300 emails and I count how many they sort the way I did. The other 150 are what I trained Leia on in round three. Every round you'll see what I got right, what it got wrong while sounding sure, the speed, and the cost. So, a bit more on each if

You don't know what these models are. Jev is from a startup called Type Safe. It doesn't write text. You give it something like an email and a question with fixed answers, and it picks one with a probability. It costs about 4 cents per million tokens and the company is reportedly in talks at a $10 billion valuation. Now, Leia is free and open source. It's 421 million parameters. It runs on your own computer and it has over 27,000 stars on GitHub. And

Just this week to open AI announced its own version, a decisions API. Now, it's only in a limited preview and they haven't said what the cost is. So, we know these type of models are groundbreaking in the future and when I get access it's going through this exact test. Now, a lot of the videos are saying Leia beats Jev showing them racing at Snake or Tetris and that measures speed, not whether the answer is actually right. And Leia's own readme admits

That out of the box it scores a 0.362 on a decision benchmark, which is worse than just guessing the most common answer every time. That's the gap I want to test on a real job. All right, so here's how I labeled them. I pulled the newest emails and went through them one at a time pressing a key for each button. The 300 test emails got locked away before either model saw a single one. Anything goes to either model, a script strips

Out the email addresses, phone numbers, links, and the sender's name, and only sends the subject, the sender's domain, and the first 500 characters. Leia never leaves my machine, and Jev is in the API, but those trimmed emails do go to TypeSafe. All right, so I put together this Google Doc, as well as this GitHub repo, that you can grab in the description below by joining our free Stride AI Academy. And if you want to run this same test with Jev or

Any other model, you can do so. I have the repo open here in Claude code, and I'm simply just going to say, "Labels are done, and the test set is frozen. We're filming. Keep what you run visible. Follow the agents.md, and round one with Jev." So, if you're running this yourself, initially you would have to go through and manually label your emails, so that it knows which ones are actually correct or not, before you run it with these models. I'm going

To go ahead and run this now. First, you're going to check and make sure that the Jev API key is in the .env file, which mine is. Of course, if you don't have that, you'll have to go to Jev's website and get it from the API platform. And now, I'm just going to tell it to go and run this command right here. All right, and we can see that round one is done, and it finished pretty quick, of course, cuz these

Models are so fast. Now, I'm going to tell it to start round two with Leia on my GPU. All right, so we can see that round two is done. It started the Leia server right here on our computer, and every email was marked as completed. All right, so after two rounds, here's the scoreboard. Now, first, the bar to beat, 2/3 of my inbox is junk. So, a lazy model that would just say junk every time would score a 64%. Jev got

A 71%, so he beats the lazy one, but only by seven points. Now, Leia, out of the box, got 22%. That's not just behind Jev, that's way worse than guessing junk every single time. So, I called 143 emails needs a reply, and only five of my test emails actually needed one. Now, the score I actually care about, did it catch what actually needs me? That's urgent and needs a reply together, which is 57 emails. Jeff caught 12, Leila caught 44, but

Only because it raised the alarm on almost everything. 156 false alarms. Here's what worries me about Jeff. Of the 52 emails I marked urgent, it called 41 of them just FYI. And sometimes it was really sure. My backup storage hit 100% full. That's urgent and Jeff said FYI. It was 97% sure. A paid sponsorship offer, someone trying to give me money, Jeff said it was junk. That's the real danger. If a model isn't sure, you can double-check it. If it's sure

And wrong, you never even see the email. Now, to be fair to Jeff, some of these are arguable. Is a failed deployment urgent or just FYI? It really depends on the day. So, I gave both models one sentence explaining what urgent means, and one sentence only goes so far. So, just like anything when it comes to dealing with AI, context is key and making sure you give your models the correct information will get you better results. Now, that's what round three

Is for. Instead of describing it, I show the model hundreds of my actual choices and Leia, it was never more than 90% sure about anything, so it can't be confidently wrong. That's not a win. It just means you can't trust its confidence yet. At Executive Stride, we run an AI and marketing agency, and we were paying thousands of dollars a month for tools that didn't talk to each other. So, I built our own business operating system, and it's called Stride Ops.

And today, our entire business runs on it. CRM, voice and text agents, workflows, content, research, all in one place. Now, I'm opening it up through the Stride AI Academy. A community with a curriculum that keeps you up-to-date with all the latest in AI. Weekly calls where I answer any questions and show you exactly how you can start and scale your own AI agency past 10K a month, or how you can master AI coding and build your own software for your business.

[music] And no matter what plan you're on, you'll get access to our own platform, Stride Ops. We're just launching, so early members will get a lot more of my personal time. And if you join now, you'll get to lock in your price as it goes up in the future. If you want to build this with me, strideaiacademy.com. Links in the description down below. Now, going to tell Claude code here to stop the Leia server and train Leia on my training emails.

All right, so round three. Everything so far was Leia straight out of the box. So, now you can see I've trained it on my own inbox using only emails that are not in the 300 test emails. So, it can't cheat. You can see here that's my 5070 doing the work right there. You can see the peak memory was about 7.78 GB of graphics memory. Okay, so now I'm going to tell Claude right here start Leia up with the train follow, run

Round three on the test emails, and open the new scoreboard. The same 300 test emails, but this time it's the Leia I trained. Out of box Leia got 22%. Let's see what it gets now. All right, and here we go. So, out of box Leia got about 22%. Now, trained on my inbox, it got 80%. Jeff got 71, and the lazy guess was 64. So, the free open source one beat the paid one. The training took 44 seconds. It wasn't luck

On a few emails, either. It got 70 right that Jeff got wrong. And Jeff only got 44 that it missed. On what needs me, it caught 42 out of 57 and 17 false alarms. Now, the catch is it basically learned two things, junk and urgent. It almost never says FYI, and it never once said needs a reply, because I only had five of those to train it on. Now, speed and cost. On my machine, Leia answered in a median of 62

Ms per email. Jeff going over the internet took 567. That's not really a fair race, cuz Leia's on my graphics card, and Jeff is a round trip over someone else's server. Now, Jeff's price is about 4 cents per million tokens. So, all 300 test emails cost me under 1 cent. That's about 2 and 1/2 cents per thousand emails. Leia costs nothing per email, and once I trained, it was even faster at 46 ms. You pay in an hour of labeling and

A graphics card instead. So, which one should you actually use? Now, if you You something working a day with zero labeling, go for Jev. It was the best one out of box and the bill is tiny. But if you're willing to sort about 150 of your own emails or whatever task you're doing, if you want to train your own model, then you know, you can train Leia. Now, on my inbox, it beat Jev after 44 seconds of training. It runs on

Your own machine and it costs nothing per email. Just know it's only as good as the examples that you actually give it. Now, I only had five needs a reply emails to train on and it never learned that bucket well enough. And whichever you pick, check the confidently wrong ones before you trust it with anything important. Now, this is, of course, just one test. Results may vary on different tests with some of these models. For the paid model everyone's hyping versus

The free rival, out of the box, it wasn't close. Jev got 71% Leia got 22. But trained on 150 of my own emails for 44 seconds on one graphics card, the free one beat it at 80%. Like I said, if you want it working a day, pay for Jev. If you're willing to label and train, then Leia is a real alternative and it's free. Now, I built the end of the guide that shows you how to run this exact test on

Your own inbox with the labeling app and the training script and the code is on GitHub, so you can point it at Claude code or code accent and walks you through the whole thing. You could also easily just tell it to customize this for a different use case and take some of the principles that we applied in this test to really anything that you're doing. All this is 100% free and you can get access to it down below in the description

In our free Stride AI Academy. Make sure to join our school community, get access to all of our resources on this YouTube channel. If you're new to this channel, we upload videos on AI from how to grow your business with AI, get clients, make money, etc. If you want to stay up to date with all the latest AI, make sure to smash that like button and subscribe to the channel. And let me know what your thoughts are in the comments down

Below if you've been using Jev or if you've been using Leia. What do you think is best for specific use cases and how are you guys using some of these new AI models? Other than that, guys, I will see you in the next video. Keep hustling, keep grinding and of course, guys, accelerate your stride. Take care.