Automated Data Engineering

I’ve built a system that can solve a traditionally tough problem in data engineering - entity resolution (are these two people / products / businesses the same) using a combination of direct LLM calls and code generation via LLMs. The direct LLM calls provide ground truth for the code generation part of the workflow and the system continuously monitors and checks the output of the generated code to ensure that it is correct. When mistakes are discovered, the code generation part updates the code based on feedback from the direct LLM calls.