This article is TBD as I’ll do a full writeup once the game is out of its current testing period, but what I can say is of all the LLM applications I’ve built this seemingly simple word game has proven one of the hardest challenges. It requires a MASSIVE network graph that until the release of gpt-4o-mini would have cost around $1000 to build in entirety. At the time of writing I’ve built, and rebuilt the engine for this knowledge graph about 5-6 times. Semantic density has proven the most difficult challenge.
Some notes from my quest to get to a high enough accuracy*:
- “Traditional” NLP still has its place within transformer driven applications
- Matching the semantic expectations of users to systems requires a multistep approach. I am increasingly becoming convinced that whilst you may not create by committee, self-regulation is best achieved through a quorum.
- One must consider the coarseness of lexical databases when using them in applications. For example NLTK WordNet is not sufficiently coarse enough to handle Polysemy as expected by human users
* = Defined as satisfying my mum who is an English Literature teacher