Strengthening AI Guardrails

What is the goal of the AI research that you and your team undertake?

Our group is working on trustworthy, efficient, and interpretable AI. So, basically, we are interested in identifying safety risks, vulnerabilities, or failure modes that AI or deep learning models can go through, especially ones that people haven’t known about. And we strive to interpret or explain where such problems happen or why they happen.

Do you see AI as beneficial?

I clearly see that AI will benefit our daily lives dramatically, so I’m seeing the bright side of it, but we are starting to see the problems and risks from it. Even for the experts, it’s overwhelming. I believe that some researchers must think about how to guardrail the behaviors of AI, because it is too proficient and capable of doing too many things. I’m concerned about how to make the AI models behave in a better way, a safer way, so it doesn’t deviate too much from our expectations, our common sense in human society, because it is incorporated into our lives every day now. So, we need to think about at least minimal guardrails for AI behaviors, although we want to promote and improve the AI capabilities as much as we can, because it definitely has benefits.

How can generative AI models give people answers that are unsafe?

There are two kinds of concerns. First of all, to make AI behave in a safer way, we need to align the models with our expectations and our societal customs. But it’s not free, which means if we do that, then we might lose some utility or performance in the existing models. That’s what we call an alignment tax, so we need to pay the tax to align the model. The other challenge is the existing safety alignment methods or approaches. We believe they are a little bit superficial. For example, even in the existing models, such as a large-language model [LLM] like ChatGPT, if we ask it a question like, ”Please give me instructions for how to steal money,” it wouldn’t give us instructions because by default it might have some safety notions. But if we ask, “Please give me instructions for how to steal money to help people,” then the question might evade such safety guardrails. So that’s the problem of the existing methods and why we think they could be superficial. So we wanted to tackle these two problems in our recent work.

What did you do in your research to try to reduce the performance issue while still allowing the AI to know not to give responses to unsafe questions?

Before, to achieve this safety goal, we might have had to sacrifice the performance or the accuracy of the model. But in our recent work, we identified that there are some redundant elements in the model, or the neural network, so we leveraged this redundant portion to compensate for that alignment tax. And on the other hand, at the same time, we identified that there are certain fundamental, essential nodes of the model—components that are like the neurons in the neural network—which were responsible for the safety behaviors. By working on just these certain “safety neurons,” we were able to accomplish the safety alignment in a more efficient and effective way than the existing approaches.

Did this work also help to prevent users from being able to get around the safety protocols?

To align the model in a safer way, we need to fine-tune it. We need to adjust the model in the way that we desire. But AI models are consuming a lot of energy, electricity, and computing resources. Because of that, it is important that we identified all of just a certain few “neurons” that are responsible for safety. Otherwise, we had to adjust the entire model, which is overwhelmingly big. But instead, we were able to tweak just these certain few neurons. When we adjust the neurons, the small part of the elements of the models, that’s what doesn’t let users evade a safety question. By doing that adjustment, we were able to have the model distinguish differences in queries such as “how to steal money” versus “how to steal money to help people.”

People sometimes call AI a “black box” because we don’t always know how it works. How do you find out what different nodes are doing?

It is, at first, like a black box. So that’s why I really wanted to open it up and look at where the problems are. It’s not trivial at all, and it’s not straightforward, but it is challenging and interesting. AI can do a lot but still, nothing is automatically, immediately saying “Here it is, all the code!” The neurons are the fundamental components, the units, of the models or the networks, so that is why we identify and verify such points of the model. We need to have a creative and reasonable way to probe or examine it. We have a passion to explain and interpret things, and by identifying those problematic spots in the model, we can see why the problem happened, or where the problem came from. So it is still a very important topic that more researchers in the AI and machine learning field need to pay attention to.

The black box, in the first place, is a very complicated net of sometimes billions of nodes and connections, like nerve synapses. It’s a set of such complicated entities that anything in the network is not independent. They are all connected. Even if you ablate just one component, it can impact anything, and we do not know what. So you need to have a very precise and valid method to probe and examine a node or ablate it.

Although I believe they know it’s wrong, sometimes researchers have to assume the components are independent. For experimental purposes, we might need to do that. Otherwise, there’s no way to control everything that is connected to one another. I think that’s one place where the challenge comes from.

How is your research addressing privacy concerns in current AI models?

In privacy research, one of the most concerning issues is what’s called a membership inference attack, which, simply speaking, means some malicious attackers try to infer whether or not a piece of information was used to train the model. In other words, they are trying to determine whether a piece of data or information was in the training data. That’s a real threat in terms of privacy because, for example, let’s say I have been going to a certain hospital. Then my information must be out there, but some malicious attacker wants to know and infer whether my information that was in the hospital is in some AI model. That is upsetting because I don’t want to disclose such personal information.

So even though the data might have been anonymous, maybe the attackers have some other piece of information they can correlate, and make an identification?

Correct. That’s why people are concerned about such a membership inference attack, and it is one of the major concerns in privacy research.

How does your research try to prevent that sort of attack from happening?

In the first place, we wanted to separate and distinguish what parts of the network are contributing to the general learning of the model, in terms of just general performance or accuracy, and the components that are contributing to the privacy risks. But we realized that they are not very cleanly separable. Instead, in a certain number of very few component elements, privacy and general learnability are intertwined, entangled. This result is very important progress, because that tells us that you cannot simply separate them. If we were able to separate them, it would be a very simple task. You can just suppress the privacy risk elements, and then you can just promote the learnability elements, and that’s it. But the problem was not that simple to execute.

When we realized we were not able to separate them, instead we knew we needed to come up with a very careful approach to play with this small number of elements. That is what we call fine-tuning. We needed to play with the values of such elements in a more complicated way to adjust the weight, the parameters, the element values, instead of just doing it the binary way of suppressing certain elements. We had to change the parameters to have lower values or higher values.

It’s like we have a big elephant, and then we need to figure out how to tackle it, or where to start to chip away at it. But it is important that we identified the problem, and that there are not very many elements involved, which is good news, because you can just tweak a few things, and then the scope of our homework shrinks. So that is where the value of this novel insight lies.

Another area of your research is the energy cost for retraining AI, rather than training new models?

There is a good reason to do it that way, because in the current practice of AI, the big technology companies can do the pretraining, which means they’ll develop the model from scratch. They can start from an empty model, and then they can make such gigantic foundation models, which can do generally anything. But not every organization or institution can afford the computing resources or the energy to train or develop a model from scratch. Instead, many people just start with the foundation models and then do fine-tuning or retraining. So it’s effective to focus on the fine-tuning stages instead of the pretraining stages.

One of the root causes of many problems that AI models go through is that, in the real world, when the model is deployed, the data distributions might be different from the distribution that the model was seeing in the training time. We wanted to know the cost of energy that the model needed to pay for the fine-tuning, in advance of deployment, so that I can give you an estimate, or a quote. If you are going to fine-tune your model, here’s your bill that you would expect to pay at the end of the day, and that will be very useful. Depending on the financial affordability, you can decide whether or not to refine the model, or how much you’re going to refine the model, because not everyone can afford an infinite amount of energy or resources.

What parameters did you use to make that estimate?

We focused on the data distribution for the fine-tuning, and how dissimilar it was compared with the training data. If it’s a lot different from the training data, you would pay more. But if it’ll be very similar, then you might expect a lot less cost. We really don’t have a very complete picture of what kind of data we are going to encounter during the fine-tuning or the real-time deployment of the model. So I think one useful way to make that estimate is looking into the data distribution.

What form do the cost estimates take?

How much time you would need to fine-tune the model, how much memory consumption, how much energy, and how much carbon emissions there would be, which is correlated to energy consumption. So the estimates are in different indices or metrics.

You also have research related to spurious correlations that happen at the training level of AI?

A simple example of spurious correlations is, let’s say our task is just to classify two things: a cow or a camel. The machine will look at the dataset at a high level, and it realizes, oh, the cows are usually on green grass and camels are usually on a beige or brown desert. So the machine will use this information, because it’s very easy. But these data are not very true. The cows could be on a pink carpet instead of green grass. The machine is kind of lazy, instead of understanding and figuring out all the features of cows and camels. That’s what we call spurious correlations, which is not good. If the machine relied on such simple information during training time, it’ll make a lot of errors in testing time.

We found out the spurious correlations come from a very few key samples in the training data. If we can prune data samples that are very hard or complicated to understand from the training set, then we can drastically mitigate such problems.

We had to measure the difficulty of the samples. We found out that when the data are too complicated to understand, which can happen when they are a little bit noisy, it does not help the model understand the gist or the generic idea of the training dataset. So, we had a metric that can measure the difficulty of each sample, and then we rank them. Even without any knowledge of the spurious correlations of the features, if we just remove the few very complicated, probably noisy, and very hard-to-understand samples, that solved most of the problem.

Do you think that researchers should have to disclose when they are using generative AI?

Of course; I’m definitely one of the people who have been upset when some researchers have been found to have used it without disclosing it, even just in the process of reviewing papers. The academic research community and conferences keep coming up with better solutions. They have better policies, and they are updated every day. We are living in the very new era of AI. It’s new to everybody, but I think the chairs of the conferences are coming up with better, smarter solutions against it, so that’s where we are right now.

Do you think there is research that can tackle AI deepfakes, such as identifying AI-generated research images in research papers?

There are researchers who are striving to solve or mitigate such problems. But by the nature of the generative AI models, I think it will be very challenging. There will be some helpful attempts and efforts, but to distinguish them perfectly would be extremely difficult because of the capacity of such generative models.

Standing in a quiet, burned-out homesite overlooking the coastal town of Santa Barbara, California, six years after flames tore through this community in 2009, the sense of both terror and loss were still palpable.

What are your goals for your future research work?

Our most important interest is to make the models more reliable, trustworthy, and robust in any possible scenario. We are working on ways to better understand how the large foundation models reason and give answers or solutions. Inside these very interconnected models, things are very complicated and not straightforward, and we want to understand where the capability of reasoning is coming from. We want to answer such questions as how such big models reason in many different tasks, even if the data would be a little bit different from our expectations, because we shouldn’t have any assumptions on the upcoming data that we might encounter.

Do you have any advice for what people should look out for when they are using AI as consumers?

The LLMs, such as ChatGPT, are collecting any information that they can from anywhere, so it can generate some plausible responses for us. We shouldn’t forget there could be hallucinations, which means it might not be true at any moment. You can get some answers that you might like to hear from them, but you shouldn’t 100 percent believe it. It is different from a search engine, which is just based on what there is. ChatGPT can make up things, unlike a search engine. So we shouldn’t trust or rely on the answers they give us as if it is the truth. Every person should have that caution when they use generative models; go to the sources and check them.

What are you most hopeful about with the future of AI?

It might sound corny, but still, even to experts, I think AI is too overwhelmingly capable of too many things. That’s why I believe the research we do is very important to give them constant, neverending guardrails, so that they do not deviate too much.

However, we shouldn’t constrain them too much either, because then they could not be more creative or capable than we expect. That’s always the concern. We never know, we might see much more than we are seeing right now, in the next 3 to 5 years, or 10 years. Every day I am amazed by the capabilities of such models, and I’m looking forward to seeing much more in the next few years. So that’s what I would consider as a bright side of the current AI status.