When I first started using Large Language Model AIs (LLM) I assumed that it was going to answer whatever I asked it nearly perfectly. While I knew it wasn't “thinking” in the same ways that humans do I expected that a tool trained on huge portions of the Internet would generally get things correct. Imagine my frustration when I ask an LLM to do a task that I understand well and it responds with something that I know doesn't make any sense.

I once asked an AI if a certain easy but likely incorrect approach to state management in React Native was a good idea. I already had a sense that it wasn't but I wanted the LLM's expertise. The LLM then changed my code to this incorrect approach. I was surprised it agreed and then implemented my suggestions. I had only asked it a question and yet it was already making changes. Then I asked the AI if the approach it had just implemented was correct and it said essentially “absolutely not” and changed the code back.

My faith in LLMs didn't erode that day but my understanding of them certainly began to change. We have a tendency to think every new technology will be a solution to every problem we've ever had; to see attributes it doesn't possess. Then we overlook its positives, amplify its negatives, and never truly learn to use it as it should be. We may abandon it entirely or simply use it to produce very sloppy work.

Try out liblab

While LLMs can't automate your SDK production, liblab can. Try us out for FREE!

There are, however, many things LLMs are good at. If we can hone in on what they're good at, and work with them, we can drastically improve the ways we work. I'll lay out the practical approach below but when working with LLMs you should keep two things in mind:

1. LLMs create an average output of text, they are not reasoning, they're giving you the average of written human intelligence. It's not always so great.
2. LLMs are most useful when they're helping you manage your cognitive load. You only have so much brain power in a day. Knowing when to do something yourself, when to augment your work with LLMs, when to use it to improve your work, and when to rely on an LLM entirely is the key to success.

A Framework for Task Automation

To effectively leverage LLMs it's essential to adopt a systematic approach to task categorization. Not all tasks are created equal, and some are better suited for LLM automation than others. A useful framework involves categorizing tasks based on their complexity, predictability, and the level of human judgment required. Tasks that are highly repetitive, and focus on already-solved problems are prime candidates for LLM automation. On the other hand, tasks that involve complex reasoning, solve novel problems, have ethical considerations, or require nuanced judgment are best left to human experts.

Implementing LLM Automation: A Blueprint for Success

When using LLMs it's crucial to design workflows that maximize efficiency while ensuring human oversight and control. A typical workflow might involve the following steps:

1. Task Identification: Clearly define the task you want to automate and its desired outcome. You must understand the task and be capable of performing it entirely yourself before you ask an LLM to do the task.
2. Prompt Engineering: Craft clear and concise prompts that guide the LLM's output. Use your knowledge of the task. Be specific; not verbose. Ambiguity and verbosity will create poor outcomes.
3. Model Selection: Choose the appropriate LLM model for the task at hand. There are many leaderboards that provide rankings for LLMs on various tasks and their rankings will change over time. In my own opinion OpenRouter's rankings are the best for everyday tasks (coding, marketing, etc.). There's also LMArena, Hugging Face, and many other sites with LLM rankings out there. They all have slightly different measurements so look for what's best for your case.
4. Output Generation: Send your well crafted prompt to your well chosen model.
5. Human Review: Review the output. Come up with changes. Be specific and ask the AI to implement them.
6. Apply Your Creativity: At some point you'll reach the limit of either the LLM's knowledge or your own. You'll have likely saved a lot of time and mental effort producing and ideating. Now is the time to take all your saved mental capacity and add your own human and creative touch.

Striking the Right Balance: The Human-Machine Partnership

The key to successful LLM automation lies in striking the right balance between human and machine intelligence. LLMs are powerful tools, but they are not perfect. They can hallucinate and generate inaccurate or biased output, and they may lack some contextual understanding. Lacking context limits LLMs because they can't learn more in isolation; they can't go searching for new information (yet) - but you can! It's crucial to maintain human oversight and intervention in this process. Humans should be responsible for setting goals, defining parameters, and ultimately, making the final decisions.

Here's an example of a recent session I had with an LLM writing this very post:

Write a persuasive blog post about the following topic: Automate the common stuff with LLMs

It may include the following topics:

• Realistic versus unrealistic expectations for LLMs. User's tendency to falsely personify AI and ascribe traits it doesn't have such as longterm memory, emotions, and reasoning.

• …

Use advanced and meticulous language. When choosing words, choose words that most precisely represent the intended meaning. Avoid fluff. Avoid unnecessary exposition.

Here's an example introduction, you may use it for theme, tone, and vocabulary. You may change and repurpose it:

Humans often have unrealistic expectations of a machine. From robotic vacuum cleaners to self-driving cars we expect them to perform flawlessly. When they fail this impossible demand we may stop using them and refuse to understand them further…

I then had a series of follow-up prompts. Then I made my edits and entirely rewrote about half of this post. Then I asked a team member to review. I still saved a lot of work, wrote what I think is a good blog post, and got some interesting ideas out of the LLM. Most importantly my cognitive load was reduced and I could funnel that brain power into the more creative aspects of this post.

The Road Ahead: Embracing Change

In summary as LLMs continue to evolve and mature they will undoubtedly transform the way we work and live. LLMs have the potential to improve the way we work by automating mundane tasks, and augmenting our intelligence. However, it's important to approach this LLM revolution with a realistic understanding of their capabilities and limitations. By embracing a balanced approach to LLM automation we can harness their power for good and create a future where humans and machines continue to work together.