Three GTM Workflows with Persistent Sandboxes

Speaker 0: Community since it started. Since a lot of people didn't know kind of a little bit of the history, I'll give, like, a very quick, you know, history. So it started about 4 years ago. There's about 12 of us in Seattle who are working with, like, very early in 2021 with, some early stage foundational models. And we Joe, who started it, really saw this as a, hey. Speaker 0: This is gonna change everything. This is really going to and he called, you know, 12 of his nerd friends, which included me. And, we got together for lunch, and he said, like, I don't wanna hear about like, I don't want pitches. I don't want anything. Like, bring your laptop. Speaker 0: Tell me what you're working on. Let's get together Ian let's share what, you know, that looks like. Let's just, like, make it raw. It doesn't matter. This shouldn't be polished, anything like that. Speaker 0: And it was really cool because, we we did that, and there's, like, 12 of us. And he Kyle jokingly was like, this is like the code brew computer club for AI, you know, for those of you old enough to know that reference. I'm dating myself. And, the, and then that kinda grew. And so the 1st meetup was, like, 12 simple, then the 2nd 1 was, like, author. Speaker 0: And then it started Buying, and, like, 1 of the main things was, like, this is not a pitch competition. It's not about polished pitches. This is about the dirty, the raw, the tinkering, the playing, the the curiosity around kind of making these demos Ian and and and working on stuff and inspiring other people to work on their stuff and sharing it with others. So fast forward. So as of today, there's about a 115,000 people in the AI Tinkers community. Speaker 0: There's over 240 cities. There's an event essentially every 8 hours somewhere in the Work, and that is happening just like you here with, coming to showcase, you know, what they're building 2026 meet other people who are building stuff. And, again, the the main ethos here is no pitches, no slide decks. This is about showing stuff that you're working on, stuff that you're building. And then this started out as kind of Clique anything in AI, anything Build. Speaker 0: And, as these models have been getting better, it's been interesting because what matters has kind of, you know, kind of shifted. Right? It's not about just creating code. It's not just about, like, Build, Butler, actually, like, how do you distribute it Ian how do you secure City? And so, go to market engineering is actually an interesting track that's kind of spawned out of the community itself being much more interested. Speaker 0: For me, it's a very special place in my heart. Like, I was an engineer that somehow became an enterprise CEO who had to learn how to sell and had to learn how to do go to Barkai, and I had no clue how to do that. And so a lot of it was just me bashing my head against the wall and figuring this out. And so now with this Track been probably the most popular track that's been, like, built around the world, Founder, like, hey. Who's building these systems? Speaker 0: Shares really are engineering systems. How do you get distribution? So it's been pretty cool around that. So really quick, this would not happen if it wasn't for CircleCI giving us this awesome space. So thank you to the CircleCI team, Speaker 1: and Speaker 0: the track itself, which is sponsored by Attio. So thank you to the Attio team. And so really quickly around this, like, how many of you it's the 1st time to a AI Tinkers event? Fantastic. How many use you knew that there was AI Tinkers outside of San Francisco? Speaker 0: Okay. That's really interesting. There web there's a whole go to market thing we need to do. Excellent. Demand who who's been here to more than who has been to more than 1 go to, AI Tinkers event? Speaker 0: Okay. 2? More than 5? Love it. Great. Speaker 0: So we got a couple of, veterans of that. So the the format here is pretty simple. We have about 5 demos right now of people who signed up. At the end of that, if you wanna demo something, as long as you're not pitching anything, you're welcome to come up. So just find me. Speaker 0: We can get you set up. Raw, dirty, anything that you've been Build. If you wanna get up on stage, get feedback. And, again, this is not meant to be polished. This is around being able to see, you know, whether people Build showcase, put yourself out there. Speaker 0: That's the whole ethos behind AI Tinkerers. So, yeah, with that said, we are going to get started with, Kyle, built, will be our Speaker 1: 1st Speaker 0: wherever he ran San. And I'll let him, take John from here. Excellent. Speaker 1: Good. Yep. Speaker 2: Alright. AI me just move everything around real quick so I can see what's going on. Tug renewal to go away. Speaker 1: Alright. 4 minutes. You got 5 minutes. Speaker 2: Yep. Alright. I'm Kyle Doherty. I'm from Attio. I'm, RevOps. Speaker 2: I run rev ops there. What I'm talking about today, is I'm gonna show you how I am migrating all our agents from other tools, into or how I'm starting to got even, migrate agents, but Date, enrichment and other tools into agents. So, and enabling nontechnical users in our GTM Work to start building agents themselves. So what I wanna show you today is, I built an Attio, agent builder, Attio agent builder, to help, help nontechnical users just build agents in Attio. We have a brand new, tool for that in Attio, and so it's super easy, to get for anyone to build whatever they want. Speaker 2: 1 of the things that I've been migrating out of data enrichment tools that we have is, an industry agent. So that's an example I'm gonna show you today. So with this tool, I'm not I'm not gonna, like, show a ton of it, but, I can just do the Attio agent Build, and it's gonna pop up, ask me a bunch of questions. Is anyone familiar with, Matt Pocock's, grill me skill? So that's what inspired this. Speaker 2: A lot of our AEs, and other people in the GTM teams, AI have a it's just hard. They don't know a lot about prompt engineering, so it can get really confusing. If they wanna have, like, structured output from the agent, it's gonna be hard. So this will ask them a bunch of questions. And then over here, I'll show you kind of what it'll spit out for them. Speaker 2: Once they tell it what they're building, it'll recommend what the agent should look like, and it'll build it in the format for the agent builder. So here, it's recommended, for me 2026 agents. So it even recommends AI, hey. You shouldn't shove this all in 1 agent. You should use 2 agents. Speaker 2: For agent 1, do a web research Agents just have it spit out plain text. For agent 2, you can have it spit out JSON. And so it'll make recommendations like that. Like, an AE is not gonna know to do that, right, when they're just playing with this. So that will help them out there. Speaker 2: So how this gets built, super easy. They just go into workflows, and they can chain agents together. So I'll go into the editor and show you real Face. I'm obviously not gonna run it too much. So I just, like, plug in the prompt. Speaker 2: And the other thing that this is gonna do, it's gonna do things like it'll have them plug in the, the Swappable at the bottom so they actually get prompt caching. It's also gonna recommend better like, what models they should use, that sort of thing. It'll stick a bunch of stuff in the system prompt, and it knows AI, hey. Put the important stuff in the system prompt so that it it adheres to that. Ian so, like, again, they're just not gonna know that. Speaker 2: Shares use could see Demand then it also does things like Mendo, like, Mendo, like, hey. You know, this is something you might wanna save to be reused Date, this 1st Prompt, because this is doing a ton of research on the company. Save it. You might wanna save that to an attribute on your company so you can reuse it later. So, again, like, these are just things they're not gonna think about. Speaker 2: So then it becomes really and the way this is built, it's really easy for people to test this out. So I've changed 2 agents here. This one's just pulling in the response from this agent. Like I said, if if I were building this for real, I would save this to the company record. And then I'm adding it to a record list. Speaker 2: Again, this is to show you how easy it is for, like, an AE to spit it out into a test. So they're not saving data to records and screwing up your CRM, and they can just go nuts here. And it's AI a safe space for them. So you don't have to have, like, a demo account for them or anything. It's, like, very easy for people to build all this. Speaker 2: Opus, like, real quick, I'll just we'll just run a few of these. So super easy. And then they can come in here, watch the runs so they can see them running. And then if they wanna check 1 out, you can see the response here. This is the research brief, but more interesting is, like like I said, you can spit it out over here, and then you can see, like, the dip like, what data has come Track. Speaker 2: And you can do things AI give me a confidence score for this attribute, things like that. And you can see here, like, this this didn't Work. So now I need to go back and iterate on City, and I would use the skill to do that in Claude Code. Ian and we Sharma to teach our AEs to use Claude Code. They're using it in desktop app, but still still works for them. Speaker 2: So it's it's been really good. So that is, the agent builder. Just a couple teaching I wanna call out. Like, I'm got be improved this, and if anyone's using it, like, let me know what you think. But simple teaching I wanna do is I wanna like, obviously, plugging these into the AI or the the UI and iterating on them gets slow. Speaker 2: Right? So I want to set it up where I can have Claude Code mimic City, loop, and get, like, you know, 70 to 90% of the way there, by giving me the out like, running the test for me, and then plug it HINTS the UI. I forgot to mention City also will suggest my skill also test, like, hey. Here's here's some tests you could run, Clique, test companies you can run. So I just throw that in the CRM and run them. Speaker 2: So that, again, they're they're doing TDD. They don't know City, but, people are just, like, doing best practices without knowing. This is, again, totally inspired by Matt Pocock. So, yeah, that's it. Other thing is, like, adding to this, I wanna add a skill that would have, enable you to build a code block DOUCOURE we have code blocks in the in the, workflow builder. Speaker 2: Same Buying, TDD, though, so you can just pressure test them before you have them go nuts on actual product data or real data. So, again, just AI getting people to do things the right way without knowing it. So, yeah, that's that's how we're using. That's how I'm enabling go to market our go to market team to build agents and test things. And then the last thing I'd say is, like, the big takeaway for me and and why I named this talk the way I did, the fact that they said Tools have a product provocative Jake, is having this it'd be this easy has made me just rethink, like, live Vercel build Prompt nontechnical, GTM leaders. Speaker 2: I used to always San, like, hey. Probably go buy and figure it out. And then if you really need to go build, then then build it. But now I think I've I've flipped that. Like, go Build, build agents, use a Work. Speaker 2: That file enable you to figure out the problem, San understand what your processes are, and then you either figure it out with workflows and some agents and some code, or you, now you know exactly what you need. Speaker 1: Okay. Thank you. Speaker 0: Stack question, which is, so you AI of alluded to it with the build versus AI. But as you're building this so 2 solutions, actually. I'm gonna ask use. 1 is you you said, hey. We Work actually gonna suggest where to split the task between 2 agents. Speaker 0: Like, how is it doing that? What is the rubric that you're giving it? AI, because I'm assuming your AEs are Speaker 1: not No. Speaker 0: They're figuring that out themselves. That's question number 1. And then the 2nd 1 is, like, what Laes been surprising when you Work building this, like, roadblock or thing that you thought was, like, interesting that you had to get around, you know, while you're building City? Speaker 2: Yeah. So this obviously isn't the production 1 that I'm building. I'll do the 1st the 2nd 1 1st. I actually I don't know if other people AI, work like this, but I find that skills AI I'm doing stuff myself are overrated. Like, I don't work in skills. Speaker 2: I just create projects for whatever it is I'm doing, and I'll set up the config. And I just can iterate way faster because I'm LLM constantly updating it. So for this, I have my own project for building these. I don't have a skill because I I haven't tailored exactly the way I want it. And so I I was able to knock the the the actual prod 1 that we have for doing, like, the industry agent for getting industry data super fast. Speaker 2: San as I wanted to, like, convert it into a skill and get it working that way, that's when I was surprised how, like, finicky it was. So that was interesting. The 1st question about how it's making those decisions, that's just from, building a bunch of Agent. Butler even before AI was doing it on Attio and figuring out when is a good time Clique, when should you split this, and just making rules for it to, decide when it gives those recommendations. Yeah. Speaker 2: Yeah. Speaker 0: Anybody else questions? Speaker 1: Yeah. Excellent. Speaker 3: AI Ian I ask you a question? So, you know, as part of these agents, you have to actually, to, like, decide what data you use to do enrichment. So how did you, when you were building this, kinda, like, decide what was the source data that you would use for for enriching records? Speaker 2: Search source data for, like, which data we want back to enrich it. Exactly. Yeah. Speaker 1: So Speaker 2: you if you were if we were building this from scratch using the, builder, I would either ask it like, I would have the AE test ask it to make recommendations and then test. For myself, when I'm doing it, I've just played with it so much that I know for specific types of data what sources I want and which sources I don't want. The other thing that I'll do is I'll also just do have Claude Code or whatever LLM I wanna use, do deep research, to figure out, like, what data I trust, what it doesn't. And then I will pressure test all those things or kick them out if I already know that I don't like them. But I honestly tackle it similar to a coding project. Speaker 2: So I actually generally use Matt Pocock's, skill, for whatever it is that I'm trying to build to find out find the best data. Speaker 3: And do you do you prefer to use Date that that has things like, you know, MCP set up? Obviously, API, Butler, like, how how do you choose them? Speaker 2: Yeah. I don't. Well, so for the for instance, with this Ian, with the I'm actually moving it all to agents and having it, for a lot of things AI industry. Like, just go find what you can on the web and from their website, to determine, like, what industry they're in, for example. And for this Ian, for industry, for example, that's 1 where I find the data is always pretty bad. Speaker 2: Right? Like, there's everyone's doing it differently from every data AI. So if you're waterfalling it, it's bad. Also, everyone's trying to cram multiple things into this 1 industry data point. Really, what you want is what are they selling Ian who are they selling Tools? Speaker 2: And then potentially a whole bunch of other metadata relative deleted to who they're selling to. So I find that using an agent is way better for that, because, AI, I can go get all that Date, and I can get it fairly cheap, and I can tailor it exactly to my business. So for instance, like, I might I don't care as much what vertical they're in. I care what are they sharing. And that's not totally true. Speaker 2: I do care what vertical they are. But for, like, initial scoring and things, that's what I care about most. Speaker 3: So Yes. Thanks. Speaker 1: Any other questions? Speaker 0: Okay. I'll ask last 1. What are you doing with your last 5 7 days of Fable? Speaker 2: I am building lots of personal projects. Okay. Speaker 0: Awesome. Date. Speaker 2: Cool. Speaker 0: Thank you so much. Speaker 1: Yeah. Track you for the show. Speaker 2: Oh, YAML. And then, so here's a QR code, if you wanna get, access to Attio. If you are a Stop and you're funded, we have, like, really good, startup, program for, like, really, really big discounts. AI go ahead and find me on LinkedIn or find me afterwards. Happy to chat with you about whatever you wanna know. Speaker 2: Whether it's API or not, you can apply everything I use or talked about today to, like, any other AI Build. So happy to chat. Speaker 0: Awesome. Speaker 1: Thank you. AI. Dexter up, we got Rob. I was Speaker 4: told no slides. We also have a Stop up program, but Speaker 1: AI Speaker 4: have a QR code because I'm not pitching anything. Actually, fun Face, the 1st time I came 2026, oh, wait. I gotta join the Zoom. I do this at the same time. The 1st time I came to AI Tinkers, it was also because it was here in our office. Speaker 4: And, about 2 minutes before I stepped up here with my slide deck, someone told me that no slide rules. I had to come up with a demo. Why do I have that's that. Tools not working for me. Speaker 5: Hold on 1 2nd. Speaker 0: We have the demo 2 minutes before, it's you go on Strategist actually a very AI tinkerer vibe. Speaker 4: So Yeah. AI it felt like I I was part of the club. Oh, there we go. Speaker 5: Sound issues. Okay. Let's see. Speaker 4: So I web all I do all day long is share screens on Zoom, but I still can't figure out how to do it, like, ever. Speaker 5: Is that okay. Cool. Alright. Let's get this thing out of the way. Speaker 4: This is not where I wanted to start. Alright. So, I wanna talk about something called, chunk sidecars that we're building here. This is chunk. So where this is test for this. Speaker 4: And we're CircleCI. All you need to know I mean, you're in our office, but we're a CICD company. We care about delivering helping our customers deliver software effectively to their customers quickly, fast feedback, etcetera. And that's shockingly changing in the world where we're all building with AI agents. So, we're working on a thing to help you get feedback right back into the agent development loop instead of waiting until kind of this later process of CI. Speaker 4: AI I'm starting with some bits of config just to talk about how this stuff works, and then I'll show you a demo of it actually, but I think this stuff's really interesting. So this is actually my AI config within this project. And what you see at the bottom, this thing called Stop. So, people may or may not be familiar with hooks within their agents. So you can basically configure your agent to take specific actions on specific events. Speaker 4: So when Claude Code thinks it's done and it's about to give control back to the user, it looks at its stop hooks and calls them. And in my configuration, what happens is it calls this command called chunk validate. And what chunk validate this is what happens if I just call that locally, without it happening inside the Agents to give you a little bit of a sense. It spins up a sidecar, which is a remote environment, Clique, effectively a remote sandbox. It syncs all of the code from the local environment into that sandbox to ensure that it's exactly right, and Consulting, and then it executes all of my tests in that environment. Speaker 4: So it looks and acts like a production or a CI environment, not like my local environment where everything is totally janky and broken to avoid the, you know, Work on my machine scenario. It also runs some stuff locally AI linting. Linting tends to fail locally and in CI in the same way, whereas tests tend to fail in very, very different ways. So I'm trying to get feedback, again, in that Tools, and it CLI occurs you know, you AI think of your CI build as maybe being, like, 10, 15 minutes. This feedback typically gets done in 30 to 60 seconds. Speaker 4: And, again, before the agent actually gives control back to me, it runs this loop. And if something breaks, it will go back to doing its work without ever telling me that it was done, if that makes sense. I mean, it'll tell me that it's doing that. But that way, I don't have to stop and run some tests and then tell it it's broken and sort of, like, resubmit that information. There's a little bit of a sidebar or a side quest on building this DOUCOURE I think this is also interesting. Speaker 4: In order to build this capability, we had to build something that detects your project, how it's structured, and how to construct an environment in which we can run all of your tests. Does it make sense? So, in order to build that, we started by building a small amount of code that could detect that for 1 product. And then we built this harness that you're seeing part of here that tries to build the environment. And if it fails to execute the test, it calls Claude, gets Claude to update the code that detects the environment, and then reruns. Speaker 4: So, ultimately, we have a very fast deterministic way of assessing your environment, but we use a set of agents to build that very fast Ian. And we just pile more and more projects into this harness. Right? I think it runs 25 projects now to make sure they can properly detect them all across 10 different sort of languages and frameworks, etcetera, using the Claude Code SDK instead of trying to do all this within cloud. Speaker 5: Right? Ian and then anyone who's working on this can add more projects and run the harness over those projects. Welcome to our office. It's always like this when we're trying to talk. Ian, so it runs it over all those projects and then use that to enhance the tool without us ever having to work on it. Speaker 5: So now I'm gonna take the bold, bold step of trying to actually run this. Trying to get something to fail is a little bit tricky, but often when I do this, it works. I'll just say that in advance. Speaker 4: It's you were just talking about Kyle was just talking about TDD. So this is AI a really perverse form of TDD where you tell the agent to build something broken, and then it'll be forced to fix it. But it'll probably predetermine that it's broken and fix it without ever hitting the stop hooks. But what we should see, is when it finishes implementing the test and says it's done, the you'll see the stop hook execution. Just to stall because all demos also involving agents take a really long time. Speaker 4: I don't know if you saw it go by, but the way that it's doing search is not setting API. I'm using something called Miro. You should check it out. It's basically, like, preindexes all of your code and uses that as a search tool instead of prepping for everything. If you, like me, are worried about your token budget, there are tools out there that can help. Speaker 4: Still shifting. Alright. Well, I if there are questions, I could take them now file this is shares this is the downside of these demos. Right? They always take, eons to execute and are always different every time you run them. Speaker 0: I I do have a question about better. So, you know, models and implement are so good with, like, flat files and API, like, speed there is great. Yeah. You're so you're saying shares better for talk. Speaker 1: Like, content the intent. Speaker 4: AI? Yeah. Around it? So indexing with a super cheap Speaker 1: model Yeah. Speaker 5: And building, CLI, and then using sort of AST style searching and stuff to say, oh, this is the thing I'm looking for. It's right here. And then feeding that information back to the model. Right? Yeah. Speaker 5: Like, from a from an outcomes perspective, I think Lead or sorry. AI could Kyle like, I don't know. It's AI any given Date, my model might choose use, like, write a Python script to search. Like, it it'll do anything. Right? Speaker 5: And those tools tend to get more precise or accurate results, probably more accurate Presenter. But the difference is small enough that the cost savings is worth it in a trade. I mean, it's something that I Agent to be trying out because I met the person who built it. And so I've there's I've met others that do things like City, but I would say, like, if if you're concerned Speaker 4: about token management, which I think Speaker 5: is, like, the conversation in every engineering room that I'm in these days, then don't test oops. Sorry. My bad. I stalled it myself. Don't just think about, like, what model do I choose, but what you feed in terms of context. Speaker 5: I think there's the project I'm thinking of is Kyle, okay. Finally, it's running a Stop Work. Okay. So you can basically, you can see at the bottom of the screen right there. I'll just explain it. Speaker 5: It's not super interesting because you know it's there in the background. We don't dump tons of information out of the screen, but it's saying, okay. Cool. I should instead of saying AI done, it's saying, cool. Now I'm running this thing in the background. Speaker 5: So, and it'll be in a 2nd say the test failed AI. But yeah. So, Headroom, someone at Netflix wrote this thing called Headroom, which basically compresses all of your text data so that, you know, you strip out control characters, you strip out anything that's extraneous because all that stuff gets fed to the LLM and gets Consulting tokens and burns, burns your cash, when usually the thing you're looking for is really small and really specific. And if you can find that thing quickly, that's a a great way to optimize. And this everything about this demo is gonna improved me wrong. Speaker 5: Fix. Speaker 1: I'm AI Date Superstore. Oh, AI, Speaker 5: I think, is the name of it. I might be wrong about that, but someone can fact check me on that 1. And it's not AI I don't think it's Netflix. I think it's just a developer at net who happens to work at Netflix who Ian search the project. Yeah. Speaker 1: Is there a reason that you chose it as opposed to just to get? Speaker 5: Because we point it to happen inside the agent loop. I'll repeat that in case anyone hear why why stop hook versus got commit. Right? Because we wanted to execute within the cycle of the agent. Right? Speaker 5: By the time I'm committing, I should probably be pretty confident in the code that I've created as opposed to, like I don't even know what's going on now. Some oh, it's it's gonna now it's trying to fix it. So which is the outcome that I wanted. I'll just let that scroll AI, though. Who knows how this is actually gonna play 2026? Speaker 5: So we want it to happen inside the loop DOUCOURE what I'm ultimately looking for, and I I think this is common, is longer times that I can step step AI Agent agent and feel confident AI I'm gonna get a good result when I come back. Right? If it's a got commit hook, then it's probably gonna stop and ask me for some other thing or say, like, hey. You know, are you engine? Whatever before like, if if I have it configured to commit all the time, then then it's gonna get triggered. Speaker 5: Right? But where would I commit that? Probably put that in a stop hook to say Ian got commit. Right? Otherwise, I'm hoping like, what we're trying to avoid is the scenario where I tell the agent, pretty Laes, when you're done building something, like, run the tests. Speaker 5: Right? Because that's super variable. And so you can't see this in here. It's totally lost. I'm gonna whatever. Speaker 5: It doesn't matter. I'm gonna blame it on the agent, not not their failing test. So, when you're kinda counting on it to just run it itself, right, then you miss it. What we have done I'm like, this is this is the nuttiest version that I've ever seen of this. I don't even know what Govit does. Speaker 5: I'm just gonna stop this. Okay. So what we have done is check for changes so that if the agent does happen to run all the tests, then we can identify Prince that last run, there's nothing that's changed. So our stop hook San just return immediately. Right? Speaker 5: But if something has changed, then it's it's always gonna hit the stop Haiku, and that's gonna get enforced. Right? So we would rather have that ensure check and do that pretty quickly. And there's still optimizations that we're working on in terms of, like, what exactly has changed? How do web? Speaker 5: We have a a different thing that we've built elsewhere called smarter testing, which basically detects which files have changed and therefore which tests to run. So, again, we can tighten that down, but we want that Presenter, and we want it as as often as possible. Does that answer your question? Speaker 1: Tools. So with with your example there, you just had, like, the agent try and fix the test that failed. Right? Do you ever have instances where a test fails because the code is broken Ian then it tries to fix the test instead of fixing the code? Speaker 5: Yes. I mean, this is, like, a pretty common scenario, I think, with agents, which is their sort of, like, use know, outcome private, AI, and my outcome is passing tests. And so we, you know, there there's, like, a combination of things that we use to try to manage that. Like, that does get back to kind of pretty Laes. We can also identify what the changes are. Speaker 5: Right? You Damaso you can set, like, a strict no test change policy. In this case, I'm trying to write new test, which do fail, but then it's kind of Clique I don't know where it's going. I wasn't paying a particular attention to it. So, yes, that's a real scenario. Speaker 5: And then we are also investing in what I would call test quality, which is tools to analyze the tests and ensure that they have not just high coverage, but use AI, the the simplest example to understand is we've implemented something called mutation testing, which is basically looking through your entire code Face for opportunities, like, basically Jake conditions or boundary conditions. Right? If I invert this sign or if I change a less than to a less than or equal to, do the test catch that to sort of determine the scope and quality of your test and then improve the test Prince then. So that's got, like, stopping that 1 Prince, but then going back over your test suite and saying, is this a high quality test suite? Right? Speaker 5: Because often the, oh, this test fails. Let me just have it return true. Right? That would get exposed in a pass of test quality. I really think, like, sort of how we thought about testing historically has been the tests are a solid gate, but that's because we Lead and understood the tests. Speaker 5: Right? And now the test themselves are subject to change based on the agent or maybe they were implemented by an agent. So vetting the quality of the test is as important as vetting better quality of the code. Yeah. Good question. Speaker 1: Pretty much. I lost you. You're using Sonic 4.6 versus 5 for the reason why you're in Speaker 4: big pocket. Oh, yeah. Because AI budget Speaker 1: oh, the question Speaker 4: was AI am I Speaker 5: using Sonic 4 6 instead of Sonic 5? I I Tinkerers is, like, my answer. This would be 25 minutes probably. I think it's a really interesting time in models. Like, 1st of all, all engineering leaders are paying a lot of attention to token spend. Speaker 5: I mean, it depends on who you are as a business, etcetera, etcetera. And so I'm constantly looking for, like, what will get the job done at the lowest price. Right? If there's a point where it's, like, spinning Ian I spent days trying to use DevStroll, I don't know if anyone's tried that, but, like, the results were comical. And then I asked Claude to fix it, and Claude was AI, like, all Engineering, like, people who review shifting. Speaker 5: This is so bad. We should just throw it out and start over Laes Claude's response. So, like, you can get very bad results from a model, but I think we're at a point certainly with some of the open weight models where they're good enough that within a tight harness right, if the harness is well constructed, you can get results that are good enough. And, like, yes, I can single shot with a fable maybe on a bigger Flask, but when I'm talking about 5 x the cost and still I have to do all the work around the outcome, then I'm gonna you know, we're constantly image the short answers. We're constantly making decisions about what what models to choose. Speaker 5: And unless I'm not getting good results, I'm always ratcheting down to kinda kinda the older thing. Good eye, by the way. I thought the font size was really small. Alright. Yeah. Speaker 5: Agent Stack, I have to get kicked off stage. Speaker 0: I I actually found John on the testing AI, so I run as a QA Agents. I run them with local models, like a LLM. Speaker 1: And Speaker 0: I I chose to do brute Face, just run every test over and over again Event with any change DOUCOURE I just got cheap tokens with the local just for QA. Right? Jake Butler agree. Like, once you write the test, you kinda have, like, Tools have a top like, a better model AI the test, Kyle of validate Anthem, and then once you just have them, just let it I I brute force it. Speaker 5: Yeah. I think I think that's exactly right. Like, I think that distributing out the work based on specific tasks and what model is necessary for that task is where we're going. But I don't want every engineer in the company to have to understand, like, when you type this sentence, then switch your model to this. And when you so you see, like, a lot of people buying or, like, deploying model routers and stuff like that right now for that reason, which is we know this class of problem can San AI can't tell if I'm, like, speaking to them. Speaker 5: Like, this class of problem can be solved with, like, Haiku. This can be solved with Kimi. This can be this needs a Face or CEO Opus or whatever. But I really do see us starting to do more of that. Again, it Demand. Speaker 5: Like, if you work at OpenAI, you're probably not doing that. But if you work in sort of AI a, you Speaker 4: know Yeah. If use, yeah, if you work San an organization with a CFO Speaker 5: and you're not the CFO, like, you're you're thinking about these things. Right? Yeah. Yeah. Speaker 1: Yeah. Thank you very much. Speaker 5: Yeah. Thank you. Thank you. Speaker 1: Jakob, GitHub. Well, I was getting up. There are 3 things we're gonna do here. We're try to run the test faster. We're trying to run Anthem minimal set of test, have those tests Date the maximum coverage. Speaker 6: Hi, everybody. My name is Jacob. I work for PromptQL, and I John talk about how the standard GTM stack, particularly top of the funnel San often feel like a nightmare Ian explain how my team and I have approached that. All GTM teams Speaker 1: have Speaker 6: context, data Lead, processes that's fragmented across 1000000 different places, CRMs, Google Docs, sales tools. And every time we are juggling between different workflows, we are constantly creating more inefficiency Ian every transfer point, and a workflow just creates more of it. So I wanted to talk about how I've compressed that down into 1 sort of simplified Agent workflow, and condensed our entire GTM stack into 3 Laes. A data layer, for my personal leads just backed by a Supabase, Ian agentic layer backed by prompt QL, and then an execution layer, which for me is outreach. So, to sort of demonstrate the workflow, I'm just gonna show, a simple automation of generating 5 LinkedIn connection requests for 5 different leads. Speaker 6: So we'll start in sort of the agentic layer here. Ever since, like, shifting away from our old stack, our entire team just lives here in this layer. We're not splitting our time and our focus between, like, a proliferation of different tools. Everything happens, with the agent. So to delegate this task, I'll just, send a prompt in that the agent knows is to, you know, do this sort of demo workflow here. Speaker 6: So a quick overview on what's gonna happen. The agent is going to Stack collect Context that it needs to execute the Work, then it's gonna pull Date from a database on the leads, then it's gonna spin up a few sub agents that will search, do the actual personalization and generation of each connection request. Then it will push it to outreach where the execution happens Ian the message actually gets sent to the lead. But the 1st step is what I wanted to spend the most time on. I think it's the most novel thing about it, and it's how this system collects context. Speaker 6: So we, store context sort of like a Wikipedia shares each, topic is its own isolated page or article in this Wikipedia. So you can see when I've, sent in this prompt here, the 1st thing the agent-led it's pulling in these individual Wikipedia pages here. And these contain, context, tribal knowledge on every single thing that makes up our systems. So it can be, you know, logistical things on how to query the database, where the data is found, message generation guidelines on, what tone Tools use, character counts, how I wanna structure my call to action, how I wanna frame messaging for senior leaders versus, you know, individual contributors, as well as all, like, the logistical glue of how to transfer between systems, push the leads to outreach, set up the email cadences, etcetera. So these, individual Wiki pages that make up our context layer are all linked to different pages, and they form this, essentially, this context graph, that makes up, like, the GTM brain for our organization. Speaker 6: And, a really cool part about it is that, like, I don't have to go in Ian author or maintain all these Wiki pages, even the ones that are only relevant to me. Anytime I'm guiding the agent, anytime I'm, suggesting an improvement or, like, flagging an error, The agent, is aware of what's in this context layer, and it will automatically per-post an amendment to this Work here. And I just 1 click it, and it will go through and make that amendment, which then becomes permanent. Ian it's also shared across our whole team. So if anyone else on the GTM team, makes an improvement to this to any sort of shared workflow, we all benefit from it, and it's permanent. Speaker 6: So it's like a, you know, Consulting, self improving asset, is what this context layer is. So to just close the loop on, like, the workflow, this is just a super base I use for my leads. Right now, it's generating for these 5, but I have thousands of leads in here. Marketing back here, it looks like it's finished, generating and personalizing the connection request. So if web check Outreach, it looks like Speaker 1: well, Speaker 6: it looks like it may have broke, but let's see here. Did it actually finish? Okay. Let me refresh 1 more time. Well, what we can all imagine would Date happened is, it's Jake those personalized test, and it's dumping them into Outreach, per lead using templates. Speaker 6: I think, honestly, the reason it's not working is I have, like, time HINTS set on here, so I'm not working past to when I should be. But, yeah, that's basically it. It's, just like a full Agent system for GTM, powered by that, like, context layer. Speaker 1: It's like a 2nd brain, but for got Tools go to market, that's cool. Question for you. When it comes to personalization, how have you noticed the effort versus impact on your outreach and response rates? Speaker 6: Yeah. I'd I'd say it's pretty big. AI, that's always the double edged sword with, you know, go to market or any outbounding is the more time I put HINTS personalization, usually, the greater the payoff, but the less volume I can do. And I don't think that trade off genuinely exists for us anymore. I think that the leads are responding to the personalization, and we're able to do it at scale. Speaker 6: So, I'd say the the effect of the personalization has been significant. Speaker 1: That's That's awesome. And then the 2nd part to that question is what type of personalization are you using, and, like, what are the signals you're using to reference within the personalization? Speaker 6: Yeah. Primarily LinkedIn. I think LinkedIn is a great source of truth for all of our own, job experiences, our interests, and stuff too, so it helps. But, yeah, the the the system was set up to take in, inputs from everything. So, some of these leads have message history dating back from when they were, you know, self serve users on a legacy product with us, like, years ago, and it's just it's able to keep everything without us having the literal or cognitive, you know, overhead of managing all the leads. Speaker 1: Hey. Cool cool demo. Just a quick question John that QL app. Is it was that AI a Work knowledge base that you had before you ever had an Agent Work, or you wrote it for the agents? Speaker 6: Yeah. So it's what's it's what's powering the whole workflow that that was prompt QL. So it it connects up to all your data, and then it also, creates and maintains that knowledge layer. It also serves as a communication layer, AI, for your team Tools that, it's picking up context and adding to that layer within the same Stop got the conversations Ian context is being exchanged. Speaker 1: I AI guess my specific question was, were you able to reuse stuff that was written for humans before and just get the agents to do the right thing? Like, you presumably had some instructions on how somebody should draft the campaign or something before. Speaker 6: Gotcha. Yeah. Yeah. Like, for me Ian my personal workflows, I had a lot of Google Docs, and that was able to just be, like, absorbed straight into that context layer. For teams that have, like, more, like, structured and well documented context and, like, confluence and other stuff like that, that can also be, like, connected up to that Work layer and and automatically absorbed. Speaker 1: Thanks for the demo. I have a question about the very end of the workflow. So you're putting your LinkedIn messages HINTS Outreach. How do you then get them from Outreach into LinkedIn? Is it are you also automating that step? Speaker 1: Because LinkedIn is pretty careful about what they allow into their system. Speaker 6: Yeah. That that Stop, I'm not automating. You're right. LinkedIn is. So, yeah, the for me, at least, the most impactful part is being able to get that individual personalization at scale. Speaker 6: So once it's San outreach, yeah, I'm doing a lot of Speaker 1: LiteLLM, copy paste? Speaker 6: Teaching. Yeah. Okay. Thanks. Speaker 1: What is it? Speaker 0: Talk, I I actually sorry. I'm gonna go through here. Get the next 1. I actually found that the the Work is a is a pattern that I've used a lot with agents myself. I actually have a shared context across, like, Claude and Laes and a bunch of other kind of, like, agents that I've built around it. Speaker 0: And, I was actually using, like, Andre Karpathy's, like, version of the Wiki on, like, flat files Ian Got, but it, like, elevated like, just a shared layer between the agents made everything better. Like, it's worth the I didn't realize that you could have it automatically build it out. I kinda, like, had to, like, manually build it. But, like, maintaining that Wiki, like, elevated, like, all the the usage. If any did anybody build a lot of content? Speaker 0: Is anybody using, like, a lot of agents for, like, building content and, like, sourcing content? That's actually a it's a really cool trick of topics. So using the Wiki behind the scenes to build out topics and conversations around logic. And as you find tweets and LinkedIn and adding to the the Wiki of those things, like, the the suggestions become really good. It was a very cool demo. Speaker 0: Thank you. Speaker 1: Web, thank you. Speaker 0: Yep. While he loads up, I'm just curious. How many people are actually working in go to market? Cool. How many people are engineers who are go to market curious? Speaker 1: AI cool as well. Awesome. Cool. Speaker 8: Hey. Thanks, everyone. Thanks, Diego, for hosting this. Of course, thank the sponsor, Search AI, Attio, for bringing us together. So, so what I'm gonna show today, it's a very actually, it's a very simplistic system. Speaker 8: So what we've done is, we have used Claude, as the main orchestrator and Build a system around it to automate, you know, our landing pages. As most of you know, landing image is very important part of your overall go to market motion. And, so as we went into, you know, the go to market motion Build that for Petavue, which is the company that I run, what we AI is that, web are kind of operating in a segment where we can ultra personalize landing pages for every account. So you can take, you know, personalized ABM to a different Scalable, mainly because, in the space that we're operating, there is far more data points that's out there, you know, on actual strategy of your prompts, their go to market strategy. So we thought it's important to actually put a very personalized landing Pages, not, you know, personalized at the industry or segment level, but at Attio real at individual company level. Speaker 8: So we kind of set out to do this, and I thought it'll be very interesting to, share it with this audience. We put a a version I mean, like, we put test most of the code out there in this GitHub repo. So, there is a Lead me file. You can point Pragun to this repo and you can easily understand what we have Ian. But I'll quickly run through, what we have done. Speaker 8: So, what we do is, Petavue use our ad optimization engine. So, we look at from 1st impression to Clique Ian, your AI, data coming in from, all of your paid DAGs. And our agents autonomous, running in the background, looking at all of the data points Ian helping you optimize every dollar that you spend, you know, in in in paid, across all of your channels. So we largely target, paid media Demand managers, growth marketers, performance marketers. That is our target ICP segment. Speaker 8: And, obviously, like most most of us in go to market Face the challenge, how do you reach your ICP, persona at scale? And for us, as I said, we are operating in a space where we kind of figured out that, you know, not just, you know, what your customer is doing, but their actual paid media strategy itself is out there in the open. So we thought we could just use that data to go very deep in terms of how we even pitch to our prospects. Right? So so that's what we set out to do. Speaker 8: And, so 2 things. So the system that we built, essentially, what it is doing is it's pulling, our PromptQL' ad data from, public ad libraries. So when companies put out ads, across, multiple channels, LinkedIn, you know, Google, Meta, Reddit, Twitter. So what ads you put out there is public information. So, you know, anyone can go out there and mine that information. Speaker 8: And in fact, if you're putting ads in, EU region, you have to, these platforms actually disclose even more information about your ads Ian this. Who is your ICP? Who you Date targeting? Who is your persona? The whole thing is out there open. Speaker 8: That is that is up for, you know, potential, vendors to actually use that information and target. And that's what we did. So the engine that we built is something that file actually mine this data, and it it uses multiple different techniques to mine this data, and use that data to qualify. So 1 of our ICP fit criteria is a company should be spending a minimum amount, in across channels, for us to be our kind of platform to be very useful. You shouldn't be just starting out with your, you know, paid media Pragun. Speaker 8: You know, our our engine, which is positioned for optimization, not just to put ads out shares. So it's not very useful. So our criteria is you should be spending 40 to $50,000 in, paid media spend every month. So we used all of this publicly available data to actually, understand and deeply analyze and benchmark marketing with our existing customers. So we know based on what kind of ads you run, what is your potential Mendo? Speaker 8: We are able to actually, estimate that. So that serves as a really, you know, a very interesting and very, very useful qualification criteria for us. And, of course, once all of that qualification is done, the engine actually goes out and builds a extremely personalized, landing page that literally reflects your that company's actual ad campaigns, actual, you know, you know, ad that they are actually running. Right? So and all of this Attio scale. Speaker 8: So pretty much, anytime I'm walking into a meeting, I'm able to easily spin up this and create these personalized landing pages, you know, before we get into a conversation. Or of course, as part of overall outreach activity, we actually, do this at scale. And if you look at the overall setup again, I'm not running this right now because it'll take AI 15 minutes to run. But again, I have run it for a few companies whose, people are actually present in this room. So I'll show you some examples. Speaker 8: AI? So, yeah. Yeah. So, and, actually I can open up the GitHub repo. So, and I can walk you through the code structure since that's kind of the focus here. Speaker 8: And shares a public GitHub repo, so I really urge all of you to check it out. Of course, you can reach out to me if you have questions. Right? So it's a it's a pretty simplistic structure. So there is a bunch of, TypeScript code, which is actually fetching data from all these different sources that I spoke about. Speaker 8: And if you look at the structure over here, this is how the overall, system is structured. Right? So, there is, you know, someone in our team is inputting, you know, what kind of companies that they John target. There is a qualification, you know, agent, you know, if AI wanna call it that. Essentially, it is Hugging, a bunch of, Node Jake Jul. Speaker 8: It is actually fetching data from different APIs to understand what kind of, advertisement companies are running. Sometimes it kind of has multiple fallbacks. Sometimes API don't return Date. So it falls falls back to using Chrome tools to actually browse, you know, ad libraries from different engines to actually fetch the relevant information. And then, there is a a verification layer built on Stop built Context it. Speaker 8: If you go and look at the GitHub, AI, you'll actually see the skills for verification. So where Claude is actually or any model for that matter is actually verifying if this information that we're getting is accurate, then it goes and builds out a landing Stack qualification and builds out a landing page. So that is how this whole system is structured. I will show you some of these pages, but this is the tool stack, right? Cloud is the main orchestrator. Speaker 8: We use browser tools, Chrome, and Playwright to actually fetch information from the, you know, AI using browser tools wherever required. The system is running on TypeScript Model code base. We use Brown fetch and phosphor, tools for, fetching brand data. Search API is used for fetching the core ad data. And, of course, on the infrastructure side, we use GitHub and Cloudflare. Speaker 8: So, essentially, even in terms of collaboration, everybody in the team actually has access to the, GitHub repo that they check out. I'm talking about everybody in the go to market T2I. So they can actually have ax they check out the GitHub repo. They actually, you know, run this, you know, Claude, Prompt. And once a landing page is created, they can push it back to a Jake GitHub repo, but a private Folder. Speaker 8: And that kind of auto gets auto deployed to Cloudflare, with a public URL with the slug. So that's how we protect company names. So, Butler, essentially, it's fully auto-generated, but still managed entirely on, GitHub structured. And, our entire go to market team is actually able to do this. So after I came down here, I just created some of these pages. Speaker 8: For example, this is the 1 that I created for Attio. You can see, you know, we Jake actually have straight reference to the actual, ad programs that they actually run across Google and LinkedIn. And the thing that I like like the most is it's kind of deeply personalized. For example, here we say that you build the CRM revenue streams trust to see every deal. You know, how do you know, you know, which of your 8 42 ads that you are running? Speaker 8: How do you know which is a City, which is actually giving you the actual impact that you actually point? Right? So if you go further down the page, it's kind of ultra personalized. I'll show you the other page that we created for CircleCI. You will actually look at this here. Speaker 8: We say that you gave engineering fast automated feedback on every build, which is exactly what the speaker was mentioning here. Right? He said that's what CircleCI AI. But again, you know, who gives your growth team exactly the same kind of feedback on every dollar that you point? So, you know, the the page is extremely personalized, not just on the ad API that people are putting out there, but also on their positioning that's available on website Ian other materials on the Internet. Speaker 8: So all of that is built through, Decompose verification architecture, you know, orchestrated through cloud and all of it's all, it's all driven by skills and all of that is available in that, GitHub repo. AI? So, so I wanted to present this, very Vertex simple and, Sharma, like I said in the beginning. Happy to take questions if any. Speaker 9: Thanks so much for the talk. Jake is Zach. I'm curious, how do you balance a personalized landing page versus the bar of, like, it goes too personalized Ian then people don't Context? Because, like, how do you know all this about me? So how do you find the sweet spot where it's personalized enough that they'll convert, but not too personalized where they go, oh, you've done all this work for me. Speaker 9: This feels like a lot, and they don't convert. Speaker 8: Right. I mean, Prompt our experience, again, I wouldn't say that we have done it for hundreds of companies yet. It's still early stage company. From our limited experience, we have seen that this level of details, people can relate to it. And, and, and again, it doesn't feel creepy because as I said Ian the beginning of this talk, all of this is public information and marketers know it. Speaker 8: People are running paid media. They know it because they are using the same information to benchmark against their competitors. So they don't get creeped out, by this. They, they very well know all of this is out shares. And, they can, at least Tools from what our experience, they're able to relate to it immediately. Speaker 8: Because when you say that agents are autonomously running your ads and optimizing, people obviously have these solutions. So what kind of optimization? What can it do that AI not doing? That's kind of the 1st question people end up asking themselves. But when they see real examples that relate to their ad campaigns, they Jake AI of able to relate to it instantly. Speaker 8: Right? That's that's kind of my observation. Speaker 9: Stacks so much. Speaker 1: Hey. Track you for the talk. So my question goes, have you did an have Date any ROI analysis on how much this improves the sales process versus, when you didn't do personalized, pages? And then the 2nd question, follow-up on that is, have you tried, comparing using AI pages without, you know, Stop Laes AI Amdashes and stuff versus, you know, Buying them on on a page? Speaker 8: Sure. In terms of RO analysis, I Buying the sample size is too small. I really don't John put out numbers. I mean, Clique, my personal feeling is that when I talk to people, there is San instant connection to this, content that we put out there. Maybe, you know, in a few months, I can share a real number. Speaker 8: AI? So, now in terms of AI Stop, I mean, Ian your question, how would we avoid AI slop? Speaker 1: I mean, AI, can we try, like, point personalized pages packages without the YAML Issues and all, you know, all the tokens that actually tell it's a a content? Speaker 8: Yeah. I mean, like we had Date, write a lot of skill. I mean, the skills are very detailed and add the verification layer on top of it. Otherwise use will eventually land up with, some kind of slop. And, I think if use, what we have seen is if we do a really good job in that, we are able to get this done even with SONNET 5, not even Opus. Speaker 8: And it works really well. I don't think the kind of content we are talking about, I Doherty think it's very sophisticated for the current quality of, class of models that we have out there. And and Event even the landing Pages, it's not too long form. It's it's a very short landing page. So, you can easily wrap around good verification skills and ensure that you can remove AI slop. Speaker 1: Question number. Yes. I'm just interested. Interest In the open Lead Date. So you SaaS search AI search api.io. Speaker 1: Right? It's AI, what kind of data exactly is that? Speaker 8: I mean, like, other than, what is not there is an easy question to answer. So, other than who you are actually targeting, pretty much everything is out there. Every ad that you per-post. When did you post? Is that today AI, not live? Speaker 8: Historically what ads you have Postgres? All of that is out there. This is all mandated by law. This is part of transparency, you know, initiated by all these, ad companies. So by law, they are kind of mandated to put all this information out there Tools know who is spending on what, who's influencing what, you know, it's test Laes basically. Speaker 8: And DAGs use said, web, if it comes to, when it comes to EU, the Laes is even more deeper. They even have to disclose who you're marketing. You know? Who is your personalized? What what is your marketing? Speaker 8: Even that is disclosed. Speaker 1: Awesome. Track you very much. Thanks a lot. Speaker 0: Is Patrick here? Patrick Damaso? Speaker 10: Patrick? Speaker 1: No? Yeah. Speaker 0: You're up. Did you get figure out how to join the Zoom? Speaker 1: Awesome. AI. Speaker 10: Oh, okay. Alright. Hi, everybody. So, from the demographic that was mentioned earlier, about half are in go to market and half are engineers. So this tool this tool addresses a little bit on the creative side. Speaker 10: Like, we've had our Claude Code moment where engineers can now code, code in their sleep, work 2 times longer because they have they have all these agents running on the background. Unfortunately, for those in the marketing team, those that are trying to design design images and, all these prompts that that translate into images, there isn't quite that automated system yet or pipeline that works so that you can go to sleep and just get everything, well help and good when you wake up. So because of this pain point, I wanted to try to design a system that does exactly this. So what it does is you start with your basic, user user prompt. It could be very simple, like a green ball John a bench. Speaker 10: Then it goes through a series of, transformations where the prompt extracts all the components that make up the shot, and then it enriches it. It provides you with a list of other arms or things that you can attach to the prompt, and it has already within that different discrete choices. And from those discrete choices, it recompiles into a natural language prompt. So you code see here shares a natural language prompt that we send to an image generation model. Now for demonstration purposes, everything here is actually local. Speaker 10: It's running on this laptop with 16 gigs of GPU. And what it does, hopefully hopefully, the Zoom doesn't screw it up. Okay. Good. So it takes your initial input, which was a green ball John a bench, and then it parses it. Speaker 10: It gives you okay. I've it has identified different engines, like the ball, the bench. There's an entity relation where the ball is on the bench, and then there are a couple of global components where you can add shot components. All of these, originated from a Date launched by Kreia about a week ago called Prompt. And OpenPrompts is just a large repository of prompts. Speaker 10: So I deconstructed or decomposed a lot of their prompts into this pattern shares you have these entities that are 1st extracted by them by a language model Ian then com comprised or or enriched by all these other layers that you add. So if you need an object, object descriptor, all of these come in, and you see that they're unset. That just means that there are so many components that you can pick from that automatically just creates variation. You can pin it down if you have things that you wanted. Let's say you wanted the ball to be made of crystal, or you could just keep it as it is. Speaker 10: Now what happens is there is a randomization. We use Attio hypercube sampling to just narrow down these choices to to or maybe about 5 choices. And then from there, we send it to the generation Model, after after about generating a couple of seeds. So you could see that there are just about 5 seeds there. So every every prompt we send generates 5 different images. Speaker 10: And what happens here is you'll start Buying, from the image generation, we're using Bonsai ternary. So it's a ternary quantized model of, of Clique flux, flux AI 4 b. And it generates it pretty quickly. So what happens is after after, after all the generations, it sends this whole batch into an encoder only model. The encoder only model, this is an image scoring model that's trained on preference alignment. Speaker 10: So what it does is it it's trained on pairs. Laes this well, between the 2, which 1 would I pick? So a higher score Ian this image scoring model means a more likely the user is gonna pick this image. And you you're gonna see that Prompt, after it generates all 5 batches, the image quality, encoder model will will come in, and then it will either reject or or accept this. As soon as it accepts this so you'll see that okay. Speaker 10: It Track actually accepted all 5. But if it rejects if it rejects an image so let me stop this 1st. If it rejects an image, it the ones that survive are also sent to a vision language model to test for prompt alignment. So we both capture the intent. Are we seeing what we're supposed to be seeing? Speaker 10: And the outcome are is what we're seeing actually shifting that we would have preferred or picked. So just just to just to recap, you you have a a threshold for generating these image. A prompt in the Model, though, those that are highlighted are all these, all these Dexter selections that change every loop. So after after a loop, you'll see some of these components change color because the the specific descriptor there gets replaced. And, the end result of that the end result of that is, basically, you get a whole Date set of what might look like an image that you would have picked that's also aligned, based on what you prompted. Speaker 10: And then the last step is is actually sending it to the humans. So when you wake up in the morning, you get this whole Date of all these images, and then you can either thumbs up or thumbs down it. It gets sent to a a dataset for positive or negative reinforcement. So that helps both fine tune the preference alignment encoder as well as the vision language model. So there you code s-eye. Speaker 10: It clicks it tags reject. And then those that are rejected are just thrown away, and then you're left with 1 image. And this image is generated by this prompt. This specific seed, structured this much Ian the image quality encoder, and it's aligned about 85% of the way to everything that you prompted. And that's it. Speaker 10: That's pretty much it. Any good questions? Speaker 0: That was that was super cool. And in particular, it's actually running on your 16 Ian CLI. I'm actually, like, super impressed with that. Just curious. Will you use the same methodology? Speaker 0: So 1 of the things that I was going with my head Lead, like, you know, this is obviously you're kind of, like, aligning around prompts and Agents, but, like, would you do the same methodology for, like, to keep on a design pattern or to, like, keep a brand language or to kind of, like, Stack? Like, would you use the exact same methodology? Or you know? Because I think that's 1 of the hardest things with, like, general image generation is how do I keep it on brand? How do I keep it aligned to a specific, like, thesis and stuff like that? Speaker 10: That's a good question. So we're working off of the we're working off of the idea here that as the models get smarter, they they stay Founder. But at the same time, preferences are still variable. Like, you could have the most AI, you'll still not pick it. So what what we're working off of the idea is as the models as these vision language models get smarter, they'll be able to pick up on your brand logos and your branding, and you can fine tune over it as well. Speaker 10: So if you mention mention your brand name Ian the actual logo is there, you can you can, you can accept that image, and it gets sent as a Date so that it it grounds the, the image generation more. And then the preference wise, because it it's also an image that you would have preferred over other generations that either had 5 6 fingers or, you know, what have use, it also stays, brand-consistent. So you have both the outcome, the output, the preference alignment, which you can train which you can AI tune. It's it's a very small with lightweight Model, 400,000,000 parameters only. It's based off of Sigliph 2. Speaker 10: And, because it's so lightweight, you scale this out to, like, an h 100 GPU. You could churn test out, like, 200 image, every 5 seconds. So you can imagine, like, what 2026 hours, 3 hours of work of this image generation looping mechanism can do, and your Date can definitely get much more curated to what you want based on based on the quality of your initial Model. So you could start with good images from strong models AI then slowly work your way down. But the system will hold Track. Speaker 10: This is almost like trying to solve the better lesson for image generation. Speaker 1: Date great. Speaker 10: Yeah. So I forgot to mention there's a learning there's a learning component to this where we're also using Gaussian processes and, Thompson sampling. What that just means is that you see you see from all of these all of these little accesses that make up 1 prompt coordinate. So imagine if the coordinate is just your prompt is just a coordinate somewhere in the latent Face. All of these different components or accesses that make up that coordinate are all these, descriptors from your prompt. Speaker 10: And you code see from here Attio, okay, if we use ball where the material is crystal, it actually has a higher likelihood of generating a good image because all 5 of those images generated an acceptable output. Whereas if we picked, from all the others and then so this is just for individual arms, but you can also see it works on pairings. Oops. So you could see that if I combine, like, a a static shot with with just light, I probably have about 28% chance of success. But if I combine eye level Ian crystal or a static shot in crystal, there's close to 92% chance of success. Speaker 10: And this keeps learning after every whole prompt search. So the prompt generates 5 different versions just so we make sure we're sampling properly, across the latent space. And then after every iteration, it updates all these priors. So you you get, you get a smarter, much more, pinpoint way of prompting for what you want. Oh, the yeah. Speaker 10: Yeah. If we use the stronger, LLM or offloaded it to the cloud, this will perform so much better and faster. But I just wanted to show because we're tinkering here. So I wanted to show that we can we can run it all locally. Tried about web. Speaker 10: It's not quite there. It's probably about 13 altogether. So 16 is probably ideal, which means you can run this on Code for free. Speaker 1: Wow. Thank you so much. Nice. Speaker 0: AI code say that just because I get to see all the demos that happen worldwide. So just give you an idea over the last 90 days in AI Tinkers SaaS been something AI 570 decompose. And, local running stuff locally has become, like, the big, like, topic of the last, like, 30, 45 days. Probably something to do with the test structure, a lot of, like, the new equipment, but, local models is, definitely something that's, like, been growing. Speaker 7: How how do I connect to Zoom? Speaker 0: Zoom. How do you connect to Zoom? You have to join the meeting. I think you have to get invited. Anthem, how do we connect to the Zoom again? Speaker 0: Sorry. How do we how does he connect to the Zoom? Speaker 1: Okay. Speaker 7: K. New meeting. Is there, like, an ID or something? Speaker 1: Yes. Okay. Speaker 7: AI, no. No. This is Okay. Can you join Speaker 1: this 1? Speaker 7: Okay. Cool. Speaker 1: So then we'll do, like, 1 2 3. That'll be the the lightning map. Speaker 7: K. Join. Speaker 11: Yep. So, 95696. 584584. Speaker 1: 071071. 07. Speaker 7: 07. Speaker 11: And then the passcode is 80000. Speaker 1: 37. Speaker 0: 1 of the things for everybody, just so you know, when you go back into, like, your orchestration, you'll get, like, a list of, like, some of the demos. Feedback is why everybody loves doing these HINTS. So please remember to go back in, give feedback. We usually, like, slice up the videos, send them to you, so it's worth, the time of, you know, doing the demo, getting, that feedback. Speaker 1: Shares. Oh, sure. Speaker 0: Use can see your face, John, so I think you're you're almost there. Speaker 7: I'm almost there. Speaker 0: Almost there. Speaker 7: I'm programmatically CLI. Not Zoom. Speaker 0: It's okay. We're gonna do AGI before we figure out Zoom. Speaker 7: LLM good. Thank you very much. Talk you very much. Good evening, everyone. My name is John. Speaker 7: I'm a classically trained computer scientist. I'm from Iowa. Just I wanna get a show of hands just to see who's in the audience. So how many people are software engineers or computer scientists? Please raise your hand. Speaker 1: Okay. Speaker 7: That's a good amount of people. And then how many people have know what push Brown automata is? Raise your hand, please. Okay. Lead me let me see if I can explain that, as simple as possible. Speaker 7: Okay. So my name is, so I work at Mendo Consulting as an Engineer, and we we we tend to have a lot of community projects where we need to design other private processor. Maybe they need a ticketing system. Maybe they need some other system that already HINTS. So oftentimes, how we did it before is just AI coding Consulting all this boilerplate stuff. Speaker 7: And that got that was fine, but then our portfolio got really big. So we had usage issues, and we we ended up with all these product, and we needed some way to do it efficiently. So I'm gonna start this demo here. And while this is running, I'll continue talking. Okay. Speaker 7: So this is going to run, and this is just going to execute the pipeline file it's sharing through this. I wanted to take a look at the, Speaker 1: 1 2nd. Speaker 7: I'll take a look at the the s-eye debugger DOUCOURE the debugger has a lot more architecturally interesting stuff we're looking at. Run demo. Okay. Let's API. So this is running a skill called dispatch 1 shot, and it takes an simple, and it executes this this little, stacks that I haven't written John Jake. Speaker 7: And all it talk describing is how to make a dog blog essentially. So if I open this up here, it's just a little GitHub issue that I think I did in JSON that describes little instructions for how you make a a dog got. Typically, how we would use it is we would go in this 1 got, and we would interview the Agent, or the Face Agent would interview us. And through that, we would create a a really in-depth specification of what we need to create that day. And then through this, this is gonna go through a pipeline called the dispatch pipeline. Speaker 7: So if I go into my GitHub directory and I go ahead and open up the debug viewer. Oh, SaaS. Reddit. This is, a little bit of a screenshot of what it's doing here. So we have the start state, and then we have a teaching phase. Speaker 7: We have an intake phase search spec. So all this is kind of within this kind of spec phase, and you can kinda see what it's doing live here. And then after spec, we have this work Ian then finally a build and then done phase. But before then, I wanna talk about kind of the theory behind this. So we have traditional, state machines here. Speaker 7: A state machine is custom simple you have a couple replace. We have transitions, and you can transition between those places. Very cut, dry, simple. That's pretty much how most applications Work. But we don't really think in this way. Speaker 7: If you if we were to have millions of these teaching graphs, we were to have Work CLI run into Issues. 1 of the issues being maintainability. How do you add more places Tools this and maintain all the transitions that currently exist? How do you how do you scale it properly to interact with other components? There's no abstraction. Speaker 7: There's no phases you can really use here. So the answer to that is using state charts Tools where you AI state within within, within Date itself to create sub state. And from here, you can create mechanisms that AI of store the configuration. And if they there's a need to transition to the super Date, the Agent application do so. So from here, if we go back to the dispatch, 1 way this this happens here is through this Build of seed intake. Speaker 7: We can either intake from GitHub, or we can do San inter interactive intake, or it can handle the job in any other way. And this we use a supersede state here so that, depending on the scenario, this can adapt very easily. And then this goes to spec. The spec does its own research to examine what we have. So here's a research loop. Speaker 7: It'll, generate questions based on the the technology we're using, based on the components that we Lead. Any any Engineering components, it'll fill in those blanks Agent continue running this loop until it's satisfied. And then it'll go through the architecture and decompose it into Work plans. The beauty of this is that it's really easy to customize for the most part. The configuration service is a little complex at the moment, but the plan is Date least to make it all YAML so that you can specify your workflow entirely within YAML. Speaker 7: And if you need to reconfigure it, it's relatively easy to do. So this is the base workflow. We have a seed. We have an intake research. These are kind of the base phases. Speaker 7: We have a budget here. Speaker 1: It's Speaker 7: about 1,000,000 engines. And we we we separate those budgets based on the different phases. And then from spec to work to build, we're able to manage the budgets, manage kind of repetitive tasks, and have some kind of reliable skeleton for formulas this will create. So looks like this is running. I don't expect this to finish, but I did run this ahead of time. Speaker 7: So before when I ran this before, it Data this dog blog demo repositories, fully coded, fully deployed to Vercel without any need for me to really, mess around with the configuration. It was already there. We this is the same test we're doing all I'm doing all all the time. But here, I gave it I I created Dexter deterministic framework so that we're using agents Ian the right places rather than giving them full control over the, code base. That's pretty much the end of my presentation. Speaker 7: The closing notes I have is that I I really wanna take a look at what happens when we get into these hyper transition states. So if we go back to the debugger, we don't really have a way of telling how how many times a research loop is going to run at the moment. What happens when an agent needs to have LLM turns and built session, interactions? Is there teaching productive that can be derived from multi turn sessions between agents, or does it just developers into something that's not useful anymore? That's all I have. Speaker 1: Thank you. Question? Speaker 0: City just actually really reminded me. I mean, I I don't know how much have you looked into the kind of, like, software pack factory patterns and, like, kind of, like, automation Ian those where, like, you have Speaker 7: Oh, yeah. Speaker 0: Yeah. Date? Like, the the pattern seems really similar of, like, scoping up front then letting the, like, mod like, essentially, like, the factory iterate in its side and have, like, some resolution gate and only, like, bubble up. And it's really easy to, like, repeat the projects that way. Speaker 7: Yeah. Yeah. Good observation. Thank you. Speaker 0: Okay. Well, thank you so much. Speaker 7: Thank you. Speaker 0: Hey. So now, we have a we we're gonna have 3 Flask presentations because they Work, like, last minute traditional. So you gotta be really tight on time. Use guys can go 1st. I think you're already on the on the Zoom. Speaker 1: Yep. Speaker 0: So you can, since I didn't have you register, you can say who you are, what you're presenting. Oh, you did. You did. Okay. Cool. Speaker 0: Okay. There you go. Never mind. Speaker 1: Alright. Let's get started. This this 1. This is the 1. Alright. Speaker 1: AI, guys. Speaker 8: My name is Rehan. This is my cofounder, Pravan. And today, we're gonna be demoing our product, Clique, as well as talking about some of the learnings that we had while building agents and agent-led systems. So for context, Click is a tool that Laes technical product founders find their audiences online, specifically on Reddit, but we're launching product content Better soon about next week. And, yeah, today, Prasanna gonna be going through some of the learnings we had while building AI agents. Speaker 1: Yeah. So I'm gonna be talking about where to use AI agents Ian, more importantly, where not to, and how we saw that while building, an agent talk platform. So the good example, which we found was writing the replies on Reddit. So CLI, how Clique works is that a Reddit post gets surfaced through the discovery process. Cheap, LLM decides whether it's relevant to the company or to the founder. Speaker 1: And if it is, it drops a reply, using the persona of the company Laes well as the founders, any style guidelines that they have, as well as making sure decompose with any Reddit rules. And it's actually native to the conversation that's happening on Reddit in general Ian CLI on that, subreddit. We have a gate afterwards to, check for AI Tools and whether again, if it's native to the conversation, a human approves City, and it goes out. And this works pretty well. So I ran, this as an example for prompt URL, which, which I can show you guys. Speaker 1: Yeah. So it found, like, a relevant opportunity for Anthem. Date a AI, that's design have any AI help. It's pretty native to what they do. And, if it's not, we can edit it, and we have a memory system that saves it. Speaker 1: Ian, yeah, the AI and rights are pretty good. And we found that Agent work well here. An example where AI agents really do not work web. While we were building Clique, Reddit removed post views from their API. And that's pretty important for our metrics and tracking. Speaker 1: And so Laes a San Aid solution, we just asked Gemini to guess the number of views that a post got, which obviously wasn't, very good. And, we quickly replaced that with a formulas. We came up with a bunch of data, AI a bunch of metadata about the Postgres of the the AI of the subreddit, how how long it's been out. Where we find that this gets really interesting is where you can use a very simple agent custom, Ian it does Work, but shares some improvements Tools be made. So Build part of Clique is actually discovering Reddit Postgres that are relevant to the founder or to to the company that they should comment on or maybe they should write themselves. Speaker 1: Initially, we did this, by just having the LLM generate, like, 30 queries that are relevant to the company or to the industry, searching Reddit Ian then classifying them. Did this work? It did work. City good. Was this cheap? Speaker 1: No. This was in 1 month. Speaker 8: When we had 2 clients, by the way. Speaker 1: When we had 2 AI. CEO web AI of had to go back to the drawing code. And rather than just kind of sticking an agent on as a solution, we kind of had to tinker around with it and, come to a somewhat better solution. And what we ended up doing was that when we onboard, a client, what we do is we, s-eye, their search queries with around, like, a 100 queries that are somewhat relevant, and we add them to a query pool. And that and then every time, discovery runs, which is, multiple times a day, it picks, 30 of them. Speaker 1: 5 of them are new. 20 of them are the top 20 ones based on a score that we AI them, and and 10 of them are random. Then it searches Reddit. Ian, again, if it's relevant, drafts 2026 reply. And then after this works, every query Face on the approval percentage that the user provides them, plus any upwards that, any, point with the previous query got, it gives them a score. Speaker 1: And then Event, we can retire the bottom 20% of queries Ian then g grow winning themes. So I can show you what this looks like again on the CLI platform. Damaso, yeah, for prompt LLM, it doesn't really have that many queries and doesn't have that much data. So it comes up with these basic queries. And then as time goes on, it's gonna AI Anthem. Speaker 1: And on the basis of real data in terms of views and upwards, it optimizes Anthem. And that's what that's what this looks like. I can show you our own personal search shares, and it's much more, it's much more fleshed out DOUCOURE we have a lot more Date, and it, like, learns from itself based on concrete signals. Stop, AI, if anybody's curious, this is what the post hoc Pages look AI. And this rules, like, the discovery runs, like, much faster now. Speaker 1: If you see it, it can, like, do it for, like, a bunch of queries in around 3 HINTS. And, it runs pretty cheaply Tools that's about it. That's that's it from us. Thank you. Speaker 0: And I have to skip through the QA because we gotta get the other 2, demos here. So, Track, thank you very much, guys. He'll be Founder, so you can ask code questions after. This is the impromptu, demo part. So folks that didn't know that they were Presenter, Strategist minute wanted to present. Speaker 9: So So be nice. Speaker 1: Always. Speaker 9: Alright. So I wanna talk to you about 3 GTM teams, things that my GTM team is doing. For context, I work at Horthy, which I'll tell you about in a sec. So test thing I think a lot of people are doing is they have a CRM. That's AI, like HubSpot. Speaker 9: What we built San internal CRM. You can see it's on Speaker 1: Hey, David. Speaker 9: Which is Date. And we still pay for, however, 1 thing that, we internally were able to do is feed all of our context into a GitHub repo, which many people have talked about. So you can see my call notes, relevant evals, and social posts, All our MD files, a lot of people have talked about this. You can see internally it's even what I post on the company versus my own personal. And what that allows us to do is have an AI internally that my growth team can go to and say, hey, I'm drafting this message. Speaker 9: What has worked in the past? And Damaso, if I send this LinkedIn DM, someone goes to my, profile and s-eye that I test posted this post. What are they gonna think? So having this view across every single social channel allows us to get much better AI of content out there. Anthem, point someone made earlier, like how do you automate LinkedIn without getting banned? Speaker 9: So we don't. 1 thing that we do is, every day so that everyone knows who, is connected with everyone else is we'll just go on control a Control CEO and just paste our connections into it. It's a neat workaround. You're not violating teaching, and this way you can take it off. You can also export your data. Speaker 9: There's test, like, a quick and dirty way that we do this. Another thing that we Founder is so right now, a lot of people are spinning up sandboxes to let their agents delegate to do a task on their behalf. We've actually found an interesting use case where, someone got my team, for example, I want them to see if I'm connected with anyone on internal LLM, internal directory of a program that I went on. If anyone's familiar with ODF or John deck, there's a program that I went on, YCS Face. There are all these internal directories that are sort of trust based. Speaker 9: You don't wanna just give your password away to simple, even if they're on your team. So what we did is we spun up a sandbox computer Ian in this, I'm always logged in. And so my growth team can go in here and just access Ian I can see exactly what they do. So instead of delegating this to an agent, I have a sandbox that's directly available for people on my team to go in and do stuff, and it's persistent. The last thing that I just wanna say in terms of an interesting thing that we could do is we're generally CLI sandbox company. Speaker 9: So 1 of the things we started doing is instead of sending links, to our landing page, we have QR codes now Ian our business cards that send Speaker 1: a Speaker 9: link to a sandbox that's spun up. So when you scan it, you're already signed into an account of our product. So you can try it for for a few minutes. And just DOUCOURE, Anthem end of the the the presentation to Shares, we're also hosting an art gallery to if you wanna do it yourself over here in a few weeks. Thanks. Speaker 0: So so fun fact, thanks to GDPR Lead no 1 better. But, you can always download all your data from all these systems, and they have to AI law. So your LinkedIn, your Twitter, all of these have they're very hidden, but you can always download it. And so 1 of the tricks that especially on the go to market side that I've always done is just have a regular, download of that, and you can keep updating your local versions of it. So, you know, you don't actually have to be, like, cruising through it. Speaker 0: So you can get all your Actions. You can get all your posts. This is required by law. So Facebook, LinkedIn, Twitter, they all allow you to do those downloads. Speaker 1: Do you have to prove that you're you have you Use based together? Speaker 0: No. It's just AI, they they it's compliance from their expensive. So, like, they have have to. So it's AI they hide the settings, but you you can always do it. Speaker 12: So Cool. AI, everyone. I'm Hannah. I'm from Daytona, and we just talked about San. AI gonna show off a demo that I've been working on, which is showing off sort of what you can do with Sandboxes, which might help with whatever you're tinkering with right now. Speaker 12: So right here, I have a bug that I'm going to reproduce. So you're doing some sort of checkout website, shopping website, and the checkout fails after you put in a coupon. And the bug appears here. Obviously, you've your coupon's been accepted, but your total hasn't decreased, so your checkout fails. And there are 3 really possible things that could be happening. Speaker 12: Like, maybe there's a front end Issues. Like, could you be accepting this too early? Maybe there's a back end issue. Like, could you be rejecting it too late? Or maybe there's a shared contract rule that's going wrong here, and there's just something that's missing. Speaker 12: So I'm going to run the demo now after we've reproduced this bug, and let me make my terminal a little bit bigger. But I'm going to pass into my I have an open AI key here Butler any agent Work, a prompt that I have a bug, and San I'm trying to test 3 hypotheses to possibly solve this bug. And I'm going to spin up some Daytona sandboxes to do this. So I have some latencies here displayed, Butler, essentially, I'm creating this parent state 1st where I'm giving it all of this context, including the bug, including the files that it might need. You can see sort of everything here. Speaker 12: Hopefully, you can see that text that's reused by the child as well, LLM of the logs that you might need, etcetera. And so now instead of just trying 3 different hypotheses, I'm doing it at the same time. I'm forking this StateCharts sandbox, and I'm trying all of these at the same AI, the front Mendo, built or hypothesis Stack end and the contract 1. Another interesting thing here is that I'm using cash tokens from the parent. You can see that most of these are, kv cache hits AI. Speaker 12: So if you're worried about your token usage, forking is a great way to get around that. And then, yeah, I made these hypothesis Work, and then it figures out that, okay. Maybe the contract 1 is the most reasonable Ian, so let's fork again. We're gonna do a multilevel fork from this sandbox. Again, same thing. Speaker 12: We're using that state, and then just continuing to pursue that bug down. And, again, there's a lot of cache hits there. And after we fork that fork, we can easily delete the sandboxes, and now our bug is fixed. So we have just a little error here that this is an invalid coupon code, and you'll never get to the weird checkout behavior. Speaker 1: Thank you. Speaker 0: Okay. And that concludes us for today. Got Event Kyle, hang out. Thank you very much. So 1st of all, thanks to all the presenters. Speaker 0: It takes a lot to go up in front Ian stage with, your working or not working or half baked or fully baked demo in in front of a bunch of strangers and put yourself out there. So big John of applause to all the presenters. Track you again to CircleCI for the food and the location. Thank you use Attio for bringing the go to market Engineering, think San San Francisco. Highly Mendo, if you find yourself traveling to another city and you feel that itch to go show something that you're building, go find your nearest AI Tinkerers event. Speaker 0: If you find yourself on vacation and you really wanna go show a demo for absolutely some reason, go find yourself an AI Tinkerers event. As I said, there's probably an event somewhere in the world wherever you're going. This is why this