Over in our Insider Chat, someone recently pointed out a tiered “REAL Rating” five-tier scale designed to replace the fairly blunt and misleading question of “did you use AI on that?” with something a bit more nuanced. As someone who believes the discussion about AI could use a lot more nuance, I’m all for the premise. I ran my own usage through their wizard and came out with a “one” — the tier for automation. While this is technically accurate, it also tells you very little about how I actually use AI.
Which is why I thought it made sense to revisit the public discussion regarding my AI usage.
But going through that thought process reminded me that it’s been a little while since I last spoke about how I’m using these tools here, and given that some readers want more transparency, I thought I’d update you on the latest. I’ve written about it at length a few times before, starting in April of 2024, when I described how I used a tool called Lex to help me edit articles. The key was always that I’d write the articles myself, and then I had trained Lex on my own writing style to ask it how it might improve the article. Sometimes it would give good advice that would make me rewrite, and sometimes… it would give bad advice that I would skip or ignore. But all in all it was a useful forcing function.
In June of 2025, I wrote about how I had built a personal task management tool, using a vibe coding tool called Lovable, which I was using to keep track of various tasks, including what I wanted to write about each day. At the beginning of this year, mostly out of curiosity, I explored whether or not you could, with a ton of scaffolding and detailed instructions and effort, get AI tools to write a passable Techdirt article, and concluded no, you absolutely could not. Every attempt was still a weak facsimile of a Techdirt article with iffy language and clichéd phrases.
In the time since then I’ve run that test for myself a few times, and even as the models have improved, in my experience, my initial analysis stands. It cannot write Techdirt-style articles well, and even if I wanted to use it that way, the rewriting, changing, and fact-checking would take way more time than just… writing it myself. So, yes, I continue to write my own articles.
What’s changed in the last couple of years is that I’ve moved away from Lex (which has not received many updates in the past year or so as the team behind it appears to have moved on to other projects), and I ended up building my own editing tool, which is built directly into the task management tool I mentioned last year (which is now self-hosted and entirely within my control). The benefit now is that I can turn any task immediately into a writing project with a single click:
That was the task that turned into this article about Judge Andrew Oldham’s bizarrely out of context quote on mail-in ballots. From there I can go into the writing project which has a bunch of useful features. Within each writing project I can add sources, which turn out to be really useful during edits. I can upload PDFs, URLs, or just add text myself (and actually, as I was finishing up this article, I had it build a new feature that will automatically add all the sources I link to in a story, so the editor checks them even if I don’t add them manually, which is another example of realizing a potentially useful feature and having it created on the spot). I can also have the tool go out and research if there are other sources I should add, though I haven’t yet found that to be that useful as compared to just searching myself.
The tool is specifically trained on my writing, and has a detailed style guide regarding my voice and style, as well as a detailed system prompt of what I am looking for in editorial help vs. what I’m not looking for (e.g., yes, challenge my assumptions and facts, but no do not think you need to rewrite what I’ve written).
After I’ve written the full piece, there’s a button to open up the “AI review” tool (which lets me use any underlying LLM engine I choose, including local ones), which will give me a critical review and make general suggestions about what works and what doesn’t. I do switch up models from time to time to see how they work, but generally will default to either Claude Opus 5 or Sonnet 5. I am also using Gemma 4 locally and am increasingly trying to use that. It’s good on straightforward pieces, but less so once the argument gets more complicated. I’m hopeful that the next generation of local models will be more reliable.
For example here’s part of the initial review of my first draft on the Oldham ruling:
The only specific line edits tend to be suggestions on typos or confusing sections, such as here where it suggested a way to clean up and tighten a poorly drafted sentence in my initial draft:
You can also see that “Discuss” button in there, which I will frequently use to debate or discuss certain suggested edits if I think the suggestions miss the mark or are unnecessary. Or I’ll just reject them if I totally disagree.
Usually at that point I’ll do a more thorough rewrite based on my own reread of the article with an eye towards what it called out as problematic. For what it’s worth, this is where the inclusion of original sources is often quite helpful, because the AI will press me on things if its review felt that I’m not being completely fair and honest regarding the source material. Sometimes it calls me out for missing stuff and suggest I dig deeper (it’s not always correct, but often is, and pushes me to reread things even more carefully). Here’s one example, where the AI editor tool told me to go back and review the original documents more closely:
It also catches me when I start rambling, which often happens on a first draft when I’m trying to figure out the best way to nail down a certain point:
Again, I don’t always follow its suggestions, but it does make me revisit and look at rewriting sections to make them clearer and tighter.
Separately, I have a built in set of “sub-agents” that will check certain things, including suggesting better headlines (this is very hit or miss, but usually at least helps me think of something better than either my initial attempt or its suggestions) and one sub-agent whose whole job is to poke holes in the arguments in the piece (this one is fun, but annoying, as it’s basically trolling myself, but in a way that helps me make the arguments stronger):
You may notice that the features I appreciate the most here are the ones that in some sense make my life harder, by telling me to work harder, check my sources, and question my arguments. Considering how often we hear about the problem with these tools being how much smoke they blow up the asses of users, it’s certainly possible to set them up to deliberately challenge and provoke you. The parts I find most useful are the parts built to disagree, because it helps make my work stronger by stress testing it before it goes out in the world.
Anyway, once the piece is then written and edited, it’s a two-click process to import the final version into Techdirt’s WordPress system and hand it off to someone else on the team for a human review before it goes into the queue to get posted.
Compared to how I described the process two years ago, the actual writing process hasn’t changed — which is kind of the whole point. The human part (the thing that is central to what I do) doesn’t get automated away. What’s changed is the structure around it. I have total control over the tool now, it’s better trained and designed to help me write better.
There are a few other more experimental features, including one where after I’ve added all my sources to the writing project, I’ll also write a short paragraph (or two or three) about my focus on the article, and the system will ask me some questions based on the sources and how I want to frame the article. This is sometimes useful in testing some of my assumptions before diving into the writing. This has been most helpful when I’m a bit undecided on what the central backbone of a piece really should be. In the past I would just start writing, realize I went down the wrong path, and then start again. Using the “editorial interview” feature has helped me a few times to think deeper about which way I want to take the article before I get to the actual writing.
I also built in a tool to also write a potential outline of the article based on that “interview,” but I’ve only tried to use that a couple times, and neither time did I follow the outline, so I’ll likely just get rid of that feature. Once again, the benefit of having total control over the tool. For all the (reasonable! understandable!) complaints of big tech companies “shoving AI into everything,” with this tool, I decide what goes where, and if something doesn’t work, I can just remove it.
I’ve also built in a bunch of other features that are simply useful for organizing my day and focus. The same tool creates a morning and afternoon briefing based on my tasks, my email, my calendar, and a growing knowledge base about me and my work. It also texts me news stories it finds that might be interesting for Techdirt (based on search criteria I have provided), which I can turn into tasks/writing projects with one response. Just last week I received my Pebble Index 01 ring that I had mentioned last year, and one of the first things I did with it is hack it so that I can speak into it to create tasks. It can also research things and leave the notes in the task tool (and text me back quick facts if I need them).
In short, as I’ve suggested repeatedly, the power of AI tools is in empowering individuals to be able to accomplish what they want to do, rather than replacing humans or doing the fundamental work. Building very personalized tools that I can constantly update and tweak to meet my needs has made me less and less interested in using software that other people built for me. It’s shockingly empowering to look at a feature that doesn’t quite work the way you want, and to be able to just tell your tools “hey, make this work differently” and have it work properly.
As I mentioned up top, using the wizard on that REAL Ratings site, I came out with a score of “one,” noting that I used AI for automation. You could argue that it maybe should be somewhere between one and two, depending on how much you consider the “brainstorming” part of what the editor feature is doing. But I’d argue this is a long way from the assumption many have of “if you use AI you’re using it for slop.”
Either way, I’m still not writing articles with these tools, and I don’t plan to start. The assumption that any use of AI means you’ve handed your brain over to it misses the part I actually find interesting: building your own tools, on your own terms, and telling them what to do rather than the other way around.