Engineering Vs. Search: What’s the difference? More from Jeremy Wertheimer

I recently flew Jeremy Wertheimer to Davos to talk about generic AI and some of the biggest issues facing today’s IT environment.

One of his most interesting points was on some of the differences between engineering and science, or rather the scientific process.

“When we make things,” Jeremis said, “we know what we’re doing… we have to do it right, we have to make sure our process is good, we have to avoid mistakes, “But we don’t have to do that by inventing new things. We just have to do the engineering… and we should get what we want.”

In contrast, he said, in science, you are dealing with uncertainties. In his words, you have to be “lucky.”

Sometimes, he suggested, we don’t know whether something is really an engineering problem or a science problem.

“People may think it’s one, but it’s the other,” he said.

Jeremy gave the example of hearing Jeff Bezos speak about large language models. Jeremy explained, Bezos suggested in a podcast that we should say that we are ‘discovering’ large language models rather than ‘inventing’ them.

That matches what Jeremy said. He gave the example of a plant – We plant a plant and nurture it, but we do not know exactly what the plant will do. We didn’t create the plant, or engineer the seed!

Jeremy mentioned smartphones, which he said would have been science fiction a few decades ago. For example, it understands what you’re saying, and it can tell you the weather, etc. This is engineering.

But then, he said, there’s the LLM: and that’s science!

Unlike smartphones, we didn’t create the LLM. We’re finding out what it can do. The nature of machine learning and AI means that some of these technologies will not be engineered in the classical sense – instead, they will be studied like biology, like a force of nature. We will study them to see what they do!

I think this is a major achievement. Building on that, in my conversation with Jeremy, he discussed a prediction he personally made – that in the future, everything will have the same three lines of code:

“Build the model, train the model, and apply the model.”

To illustrate, he gave the example that epitomizes so much of the progress that has been made in technology over the past few years – the toaster!

When you think about what a toaster does, you might consider metrics like bread moisture, heat, and other factors — but at the end of the day, if you don’t proceed with a completely deterministic approach, you Don’t do this. I don’t know how the model works – not completely.

Anyway, Jeremy, who minored in neuroscience while researching AI, also made the analogy of the human brain.

“Brains are very complex,” he said, adding that today’s LLMs are also becoming more complex, with more neurons, and eventually, they will defy easy dissection. He talks about the phenomenon of training versus building and our expectations for LLM, calling the realization of our limitations a “bitter pill” – in that we will eventually learn that we can’t always figure out how or why a model works. Does something.

I feel like that’s a really enlightening way to look at technology. We will either know how a system works, or we won’t. We’ve talked a lot about explainable AI at conferences and in the media. And this general idea is that we have to be able to keep AI explainable. But Jeremy’s claim kind of contradicts this, or at least points to a certain kind of different perspective: that we have to settle for less than a complete understanding of how complex models work.