Users in the Loop—Preparing Agents for Work in the Real World #SFTechWeek

Thursday, Oct 8

5:00pm

Hosted by Dscout

ᵔ▿ᵔ

˜◡˜

◉‿◉

Models learn in sterile environments, but fail when people use them in the real world. The gap is the long tail: open-ended, non-verifiable work that benchmarks and small expert panels were never built to define, and that only surfaces when real people do real tasks. The signal that closes that gap is already being generated. Real users correcting agents—voice notes, screenshots, written notes from actual work—produce demonstration and preference data no expert panel or crowd platform can manufacture. Users in the loop, not humans in the loop, is the next frontier of post-training data. Over a private dinner, 10-15 researchers, product leaders, and founders across post-training, evals, and agent deployment will dig into where the long tail actually lives and who can generate that signal at scale. You'll leave knowing which agent failures matter most, where the feedback to fix them comes from, and who's already capturing it. Hosted by Ground Truth, Dscout's practice for consented, in-context human data—demonstrations, preferences, and evaluations for training and evaluating AI. Every participant is ID-verified and opted in, and data lands continuously through a programmatic API, so what you build on traces back to a real person, not a model checking another model's homework. Format: invite-only dinner in a private dining room at a San Francisco restaurant, 10-15 guests, on October 8th. Venue will be shared with confirmed guests. This event is a part of #SFTechWeek—a week of events hosted by VCs and startups to bring together the tech ecosystem. Learn more at www.tech-week.com.