Humans-in-the-Loop Agents Hackathon

WELCOME!

Thank you for participating in the AI Tinkerers Humans-in-the-loop Agents Hackathon, sponsored by Google Cloud at Weights & Biases. This guide will help you get started with building human-AI collaborative systems using the latest tools and frameworks.

The goals of this event are:

• Learn about human-in-the-loop AI architectures
• Build practical systems that combine human and AI capabilities
• Connect with experts and fellow builders
• Push the boundaries of what’s possible with human-AI collaboration
• Leverage Google Cloud’s Vertex AI & AI Studio in your project

How Judging will Work

The preliminary round of judging will take place in two rooms in divide and conquer style. Each team will play the video they submitted and judges will have a chance to ask a couple of questions and then they will place their votes. This will maximize the number of votes per team and give a chancesfor teams to present in front of a large number of attendees

Based on this, the judges will tally the results and select 7 finalists. The 7 finalists will present in front of the all attendees, judges and guests on the main stage in a live demo format – 5 minutes each.

GETTING STARTED WITH HUMAN-IN-THE-LOOP AI

Core Components

To build a human-in-the-loop AI system, you’ll typically need these components:

1. AI Agent Layer

• LLM for reasoning and task execution
• Specialized skills or tools the agent can use
• State management for multi-step processes

2. Human Interface Layer

• UI for reviewing agent outputs
• Controls for correcting or steering agent behavior
• Clear visibility into agent state and decision-making

3. Orchestration Layer

• Logic for when to involve humans
• Feedback incorporation mechanisms
• Context management across interactions

Key Principles

When designing human-in-the-loop systems, consider these principles:

1. Transparency: Make the agent’s reasoning visible
2. Controllability: Allow humans to intervene at key decision points
3. Learning: Incorporate feedback to improve future performance
4. Efficiency: Automate repetitive tasks while preserving human oversight
5. Trust: Build systems that earn user confidence through predictable behavior

TOOLS & RESOURCES FROM SPONSORS

All participants will leverage Google Cloud’s Vertex AI & AI Studio for their projects, and will receive credits and support prior to the event. Experts from Google and other sponsors will be on hand to support you to build and deploy state-of-the-art agent systems easily.

For more information on obtaining your credits, please head over to the sponsor section of the hackathon website.

PROJECT IDEAS

Here are some potential projects to consider:

1. Vertical Semi-Autonomous Agent

• Build an agent for a highly specific vertical using your favorite agent framework
• Build a full-stack app around this vertical use-case
• Integrate this agent into an application (potentially using CoAgents by CopilotKit)
• Allow seamless collaboration between the human and the agent using the application

2. Investment Portfolio Canvas

• Build multiple agents around the different use-cases of a portfolio app (Research agents, buying agent, etc)
• Integrate these agents into a full stack app and use something like CoAgents as a router to choose the right agent at the right time
• Share state between the full stack app and the agent so the human & the agent can seamlessly collaborate

3. Home Helper Agent [Advanced]

• Integrates with smart devices in your home, has access to the user’s calendar, etc and helps the user plan their day and optimize their action
• Uses agents to execute on multi-step operations such as booking appointments, sending messages to friends, etc
• Tip: Human-layer to verify actions before taking them

4. Marketing Agent Application

• Use open-source social media posting apps
• Build Agents that help craft social posts in the style of different account
• Build agents that can execute social media posting
• Combine the agents with the full stack app so that users can collaborate with the agent and correct its output

5. Legal Document Assistant

• Design an agent to assist in drafting and reviewing legal documents
• Lawyers can interact with the system to modify suggestions
• Share document drafts seamlessly between the agent and lawyers
• Include an interactive Q&A for specific legal inquiries

6. Social Media Campaign Planner

• Agents draft posts and suggest timing based on the user’s network
• Marketing teams review and modify content
• AI adapts to brand voice based on team edits
• AI deploys content based on human-in-the-loop suggestions

Other tips: Gemini is especially good at video understanding and long context in general, so ideas that uniquely leverage those aspects might be interesting.

JUDGING CRITERIA TIPS

Projects will be evaluated on running code (25%), innovation (25%), real-world impact (25%), and theme alignment (25%). For detailed scoring rubrics and evaluation criteria, please see the Judging section on the hackathon website.

Rules

Must Have:

• Use Google Cloud’s Vertex AI or AI Studio in your project
• Use Google Gemini in your project
• Include a human-in-the-loop component
• Have 2-6 people on your team
• Participate in person - no remote participation allowed
• Clearly document your starting point:
  • It’s OK to build upon existing projects
  • You must clearly mark what existed before vs. what you built during the hackathon
  • Only work done during hackathon hours will be judged

Nice to Have:

• Additional AI models/tools from other sponsors present
• Clear attribution for any tools and services used

Don’t:

• Submit pre-built projects.
• Create harmful applications
• Share other teams’ private code or ideas without permission
• Use copyrighted material without permission

Questions? Ask in Discord or find an organizer

Event Logistics

Location & Venue

The hackathon will take place at Weights & Biases headquarters in San Francisco. We strongly encourage all participants to work from the venue to take full advantage of the resources available, including direct access to Google Cloud experts and our other sponsors who will be on-site to help you build amazing projects.

Schedule

The official schedule and key deadlines are available here.

Food & Refreshments

We’ve got you covered throughout the event:

• Full breakfast, lunch, and dinner served on Day 1
• Breakfast and lunch served on Day 2
• Continuous coffee, beverages, and snacks available throughout both days
• Special dietary requirements accommodated (please indicate any restrictions during registration)

Resources & Support

• Google Cloud experts and other sponsor representatives will be available on-site throughout the event
• Visit the event website for:
  • Sponsor credits and resource access
  • Detailed judging criteria
  • Team formation tools
  • Technical documentation
  • API access and setup guides

Project Entries

➡️ The deadline for project submissions is Sunday at 1:30 PM.

➡️ Please familiarize with the judging criteria

➡️ Your project must make use of Google Cloud’s Vertex AI or AI Studio.

➡️ You must submit a 2-minute video of your running project uploaded to YouTube for the judges to review. This short video must show the running code and include any explanations to help judges evaluate your project.

➡️ You can submit a project you started working on before the hackathon; please just be clear in your submission what you did this weekend (versus what was already implemented prior to the event).

➡️ You retain all intellectual property rights (etc, etc) to everything you work on during the hackathon and submit for judging. Anything you do submit for judging might get posted to social media, though! (So don’t submit anything that you want to keep secret.)

Event video and photographs

At various times people will be taking photographs and recording video for their projects or for social media. Please be aware that you may appear in video and photographs that are posted to social media.

Notwithstanding that, nobody should take photographs or record video in any situation where someone has a reasonable expectation of privacy. Please be respectful. See the Code of Conduct below.

Important Notes

• Take advantage of being on-site! Our sponsors and mentors are here to help you succeed
• Check the event website regularly for updates and resources
• Don’t hesitate to ask questions - we’re here to help!

CODE OF CONDUCT

We’re committed to maintaining a welcoming, inclusive environment:

• Treat all participants with respect
• Work with integrity and honesty
• Focus on constructive feedback
• Help others learn and grow
• Follow sponsor platform guidelines
• Report any concerns to organizers

Questions or Concerns?

Find an organizer at the event or reach out through our Discord channel: #sf-hitl-agents-hackathon

GETTING HELP

Visit sponsor booths for technical guidance
Connect with mentors in your area of focus
Join the event Discord # sf-hitl-agents-hackathon: https://discord.gg/PCUJNK7z

This guide is licensed under CC0 Public Domain. Feel free to adapt and share!