Remarkable facts about AI that I learned after taking nine courses on LLM
This year, I completed certifications from Anthropic, Microsoft, LinkedIn, the Project Management Institute (PMI), and the National Association of State Boards of Accountancy (NASBA).
Here is what I found interesting from them:
1. "AI" is actually NOT a strict scientific term. It is just a commercial buzzword that does not fully survive academic audit. The problem is the second word: "Intelligence".
According to a scientific consensus statement signed by 52 researchers (L. Gottfredson, University of Delaware, 1997)1, real "intelligence" should have 7 features of thinking. And AI - even the most advanced AI - does not meet all 7. Only 5 of them have been proven.
Intelligence is a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.
Linda GottfredsonPsychologist, University of DelawareConference photographSo we have not reached true "Artificial Intelligence" yet. What is currently on the market would be more accurate to call "Algorithmic Modelling," but that does not sound sexy and is harder to sell.
2. The hardest part of language for AI to process is the pronoun.
The problem is that pronouns themselves do not mean anything in language. They only point to something.
That is why, when a neural network's multi-vector attention architecture2 sees a pronoun in a text, it starts spinning its "head" like crazy, trying to understand what exactly the pronoun points to.
In English, the worst word for LLMs to understand is "it."
Based on this phenomenon, researchers developed a special test for machines' understanding of pronouns: the Winograd Schema Challenge3, which AI systems could not beat for 8 years4.
3. Five years before Sam Altman, Google developed ChatGPT-level technology, but completely failed to understand what kind of "bomb" it had invented.
After playing around with its AI chatbot, called LaMDA5, Google placed its greatest invention in the archive and did not show it to anyone.
Years later, OpenAI built something similar, following Google's instructions. ChatGPT revolutionized the economy and created a new market, leaving Google behind.
References
- Gottfredson, L. S., Mainstream Science on Intelligence: An Editorial with 52 Signatories, History, and Bibliography, Intelligence 24(1), 13-23 (1997) - https://doi.org/10.1016/S0160-2896(97)90011-8↩
- Vaswani, A. et al., Attention Is All You Need (arXiv:1706.03762, 2017) - the transformer architecture, published by Google - https://arxiv.org/abs/1706.03762↩
- Levesque, H. J., Davis, E., Morgenstern, L., The Winograd Schema Challenge, Proceedings of KR-2012 - https://cdn.aaai.org/ocs/4492/4492-21843-1-PB.pdf↩
- Trichelair, P., Emami, A., Cheung, J. C. K., Trischler, A., Suleman, K., Diaz, F., How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG (arXiv:1811.01778, 2018) - on dataset artefacts in Winograd-style benchmarks - https://arxiv.org/abs/1811.01778↩
- Thoppilan, R. et al., LaMDA: Language Models for Dialog Applications (arXiv:2201.08239, 2022) - https://arxiv.org/abs/2201.08239↩