Most organizational data is unstructured, and LLMs make it easier to use that data for domain-specific applications.
2
The work around the model is larger than the model code itself, including data access, deployment, evaluation, infrastructure, and governance.
3
Retrieval-augmented generation can address problems such as stale knowledge, weak attribution, and hallucination, but it introduces its own engineering problems.
Summary
Ahmed Menshawy describes how AI work is moving from structured data toward LLM applications that use the unstructured data held by organizations. He argues that current systems should be judged by the risks they create today, rather than speculation about AGI or future doomsday scenarios. LLMs can augment human productivity, but their autoregressive design can amplify errors and does not provide a path to general intelligence by itself. The main engineering burden lies around the model. Teams need model access, customization environments, application tools, scalable inference, data controls, evaluation, and governance. Menshawy compares using a foundation model directly with coupling it to external memory through retrieval-augmented generation. RAG can improve factual grounding, freshness, and attribution, although production systems still need careful retriever and generator design. Mastercard's approach combines adoption with safeguards for privacy, security, reliability, access control, and bias.
LLMs open organizational data that has long been difficult to use
Menshawy begins by contrasting recent AI progress on structured data with the data most organizations actually hold. He says more than 80% of organizational data is unstructured, while 71% of organizations struggle to manage and secure it. LLMs make it easier to use this material as contextual information for language models. An organization can provide domain-specific data so a model formulates answers with that context. This changes the practical AI problem from labeling structured records toward building systems that can retrieve, control, and use documents and other unstructured sources safely. The benefit depends on how well the organization manages the data, since the same information must remain secure when it is connected to an LLM.
Current AI risks deserve attention before predictions about AGI
At Mastercard, Menshawy describes generative AI as a way to augment human productivity. He pushes back on claims that it will inevitably replace jobs or cause a future catastrophe. Citing a Nature article, he says attention should move from speculation about tomorrow's AI to risks already present in current systems. That framing also gives regulators a clearer target for laws and policies. He argues that rules can address existing harms while remaining ready for new algorithmic approaches. The point is practical: organizations should assess the systems they are deploying now, including their misuse and failure modes, instead of basing decisions on predictions about what AI may become.
Autoregressive language models can amplify their own mistakes
Menshawy says the algorithmic foundations behind current LLMs will not by themselves take us to AGI. He refers to Yann LeCun's discussion of objective-driven learning and explains that an autoregressive model depends heavily on what it has already generated. When it makes a mistake, later token generation can build on that mistake and amplify it. He connects this limitation to Ada Lovelace's 1843 observation that the analytical engine can only do what people order it to do and cannot originate anything by itself. In his view, LLMs can transform applications while remaining limited at their core. Their useful behavior should not be confused with independent understanding or invention.
ChatGPT changed the interface around an older language-model idea
Menshawy says next-token prediction is a simple idea that is decades old, and OpenAI did not invent language models. He identifies the major change as the user interface. ChatGPT made it possible to prompt a model naturally and receive a response in the same conversational style people use with one another. The base models were trained on internet-scale data, but the conversational behavior required later work. Menshawy describes the use of outsourced workers to create manual question-and-response pairs before the release of GPT assistants and ChatGPT. This distinction helps separate the underlying prediction method from the product experience that made LLMs accessible to ordinary users.
Production GenAI needs infrastructure that older ML tools do not provide
Menshawy lays out four requirements for a GenAI application: access to several foundation models, an environment for customizing contextual language models, an easy tool for building and deploying applications, and scalable ML infrastructure. He says access to models is comparatively manageable, although teams still trade off model size and cost. Enterprise environments are harder because they were not built for such large models. The application-building tool is the hardest part in his assessment, since the tools used before GenAI were not designed for LLM applications. Inference infrastructure also has to scale replicas quickly for end users. He cites OpenAI's curve showing inference GPU and RAM demand growing beyond training compute.
Most AI engineering work sits outside the model code
Using a 2015 NIPS paper, Menshawy says ML code makes up less than 5% of an end-to-end machine-learning pipeline. The rest includes the surrounding systems needed to collect data, deploy models, operate them, and connect them to products. He rejects the idea that an AI engineer mainly connects APIs and handles plumbing. For LLM applications, the surrounding work includes model adoption, domain data, infrastructure, and controls. His own paper presents a similar picture for LLM systems, where the boxes around the LLM code account for more than 90% of the application. The model is only one part of the production system, so technical debt accumulates in every surrounding component.
Direct foundation-model use leaves several production problems unresolved
Menshawy describes a direct foundation-model approach that can use zero-shot or few-shot prompting, or fine-tuning with domain data. Enterprise teams still struggle to operationalize these models because of accuracy requirements. He names confident hallucinations, weak attribution, stale knowledge, revision requests, and limited customization. A model can become outdated as new releases appear. People may also opt out of AI systems under rules such as GDPR or California AI law, which creates a need to edit the model or stop information from influencing decisions. Fine-tuning does not make these tasks easy. Teams also need the model to stay grounded in their own domain data rather than produce unsupported general answers.
RAG adds useful controls, while joint retriever-generator design remains difficult
Menshawy presents retrieval-augmented generation as a way to connect a foundation model to external memory. Domain-specific context can improve factual recall, and vector indices can be swapped to keep information current. Retrieved sources also provide a path toward attribution, since teams can inspect why the model produced particular text or decisions. He cites a paper on augmentation reducing hallucination in conversation. However, he says production RAG raises many unanswered questions, especially how to optimize the retriever and generator together. Common systems treat them as separate components with no awareness of each other. The original RAG paper instead trains them in parallel, which requires access to model parameters and allows the generator to learn from retrieved information.
"The solution to all of these problems is really to couple the foundation model to an external memory, also known as the RAG."10:49
Who should watch
You are building an LLM application inside an enterprise and need to understand why model selection is only a small part of the work.
Your team is considering RAG and wants a practical list of the production issues around freshness, attribution, access control, and model coordination.
You are responsible for deploying AI under privacy, security, reliability, or bias constraints and want Mastercard's framing of responsible adoption.
