Just copying over some notes from a great video, in case it ever goes away:
Site Analysis and AI's Strengths and Limits
By David Hobbs
Progression of content analysis
| # | Type | Example | Automation at scale? | Use Al? | Quantitative or Qualitative? |
|---|---|---|---|---|---|
| 1 | Traditional Automation | Crawl with [[Screaming Frog]] | Yes | No | Quantitative |
| 2 | Manual | Line-by-line qualitative audit | No | No | Qualitative |
| 3 | One-off Al | Ask ChatGPT a question about a site | No | Yes, one time | Qualitative |
| 4 | LLM → Deterministic | Have Claude write a script to run reading level analysis on every page | Partial | Yes, one time | Quantitative |
| 5 | Scaled Al | Pass thousands of pages to low power LLM to categorize | Yes | Yes, every page | Qualitative |
| Note: Al allows us to do some qualitative analysis that is NOT manual. |
Advantages of particular types of content analysis
| # | Type | Advantages |
| 1 | Traditional Automation | Easy to get basic information on many pages. |
| 2 | Manual | The approach everyone knows, and it feels like things are being done in a high quality manner (even before Al this was often the feeling). |
| 3 | One-off Al | Easy to get analysis that might be outside your expertise for a subset of pages. |
| 4 | LLM → Deterministic | Only use Al once, and then have all the advantages of traditional automation. |
| 5 | Scaled Al | Get qualitative information over a large swath of content (possibly impossible otherwise). |
Disadvantages of the manual approach
- Almost certainly leads to restricting the number of pages that are analyzed.
- Very difficult to iterate or change course (such as if you decide to change the definition of audience)
- Very difficult to merge analysis from multiple people.
- Very difficult to be consistent.
- Difficult to track over time.
- Likely to encourage team to take action on content while analyzing it.
Disadvantages of one-off Al
- Generally low transparency (How did it come up with the answer? Did it look at any pages? Did it consider your existing analysis?)
- Related to the above, it can definitely hallucinate or over-extrapolate.
- Generally requires powerful LLM
- Can short circuit critical thinking of the analyst
- NOT for pervasive analysis across a digital presence
Tip: If you do one-off chatbot analysis, explicitly ask how it came up with the answer and its sources. ie., "Explain to me in detail how you determined this. What sources did you use? What did you derive from each source? What confidence do you have in this answer?"
Disadvantages of LLM → Deterministic
- Depending on skillset, can be difficult to evaluate whether the script is doing what you want it to do
- Can be difficult to clearly specify the requirements
- One-off efforts to write scripts can be difficult to reconcile together into a coherent whole
Keep in mind non-LLM automated analysis options
- Near-text duplicate analysis
- Graph database analysis (analysis of linkages between content / sections)
- Scraping out information and component analysis
- PDF analysis
- Merge information (for instance from Google Analytics)
- Reading level analysis
- Charting
- HTML structure analysis
- Over-time analysis
- Logo analysis
- Formulas, maps, and rules
- Technology analysis (such as with wappalyzer)
Tip: Consider converting problems to deterministic ones with AI
Disadvantage of Scaled Al
- Difficult to develop on your own
- Heavy processing is in the backend (not in chatbot)
TIp: Clearly differentiate between what's EASY to analyze and what's IMPORTANT to analyze.
Note: Far too often teams just revert to what's simple to analyze when that will not help them achieve their real business goals. For example, you can't analyze for consolidation if you only look at a small subset of pages because then you would be able to use a spreadsheet.
Summary
- There is lots of automation that can be applied outside of LLMs.
- For some questions, you really do need to look at the entire digital presence.
- Al can help you accomplish qualitative assessments that required manual intervention before.
There are a variety of ways of using Al for site analysis.
Tips
- Focus on refining the QUESTIONS you need answered.
- If you do one-off chatbot analysis, explicitly ask how it came up with the answer in its sources.
- Consider converting problems to deterministic ones with Al.
- Clearly differentiate between what's EASY to analyze and what's IMPORTANT to analyze.