NVIDIA Project G-Assist Hands On and Under the Hood

[btarunr]

On Sunday, NVIDIA announced Project G-Assist, the AI chatbot for gamers that can be pulled up in the middle of the gameplay, and sought help from. You could just pause your game and Google for help, but G-Assist can be activated in game, and is situationally aware of your game (e.g.: it knows where you're stuck and how to help you out). It won't play the game for you, but give you actionable info on how to play, or improve. For example, you could ask it how to craft a particular weapon, and where to find the items needed in game, and get concise guidance. G-Assist also knows about your graphics card, framerates and other telemetry data.

We went hands-on with G-Assist, and found that it's very capable of doing the things NVIDIA claims it can, short of playing the game for you. They are showing a demo of ARK Survival, but the impressive part is that there's no integration of G-Assist in Ark—rather is runs as an injected overlay that can capture user input. This means that G-Assist can work in ANY game, even without official support. We also learned how G-Assist works under-the-hood, particularly how the chatbot is situationally aware of your game, and it's fascinating.




Besides an AI model to recognize spoken text, it runs multiple computer vision models to understand what's going on in-game. The first one is an OCR model that recognizes text on screen—like mission objectives, NPC names, etc. On top of that, another model is used to recognize objects in game—like enemy types. Since it's "seeing" your gameplay, it can tally what it sees with its vast pre-trained model of information, to come up with answers that are tailored to the last word. NVIDIA says that the performance cost of having G-Assist running is very low, since unlike DLSS, the application isn't sitting inside the graphics rendering pipeline, it's leisurely seeing frames the way a screen recording/streaming software would, and runs the compute-intensive model operations only after you give it a query to answer. To achieve that it keeps a log of previous frames that it encounters and only analyzes them when necessary.

Besides being an assistive AI, it can also take actions to make the right graphics and game settings. For that it is integrated with GeForce Experience Optimal Settings. For example, and NVIDIA demonstrated that working live in the demo, you could tell it to improve your framerates, either by overclocking, or changing details settings. You could also ask it to enable DLSS, or undervolt the GPU. Since it has access to live telemetry from the GPU, you can also request a chart of latency, power usage or GPU load. NVIDIA made it clear that this is a tech demo that's designed to show game developers what's possible in games if they integrated an AI-powered assistant.

View at TechPowerUp Main Site

 

[Onasi]

Uhhhhh, this seems… borderline cheating, if that works as demonstrated in reality?
Like:
You could tell it "hey I wish this button performed this combo," and it would configure things accordingly—everything short of playing the game for you.
That’s an effing yikes for competitive integrity of a lot of fighting games, for example. There are moves that are deliberately limited by execution and just putting them on a button when they aren’t SUPPOSED to be is… yeah.

 

[Ferrum Master]

I can see VAC and others banning this.

Also the impact on latency, stutter... It will not take off.

 

[btarunr]

Uhhhhh, this seems… borderline cheating, if that works as demonstrated in reality?
Like:
You could tell it "hey I wish this button performed this combo," and it would configure things accordingly—everything short of playing the game for you.
That’s an effing yikes for competitive integrity of a lot of fighting games, for example. There are moves that are deliberately limited by execution and just putting them on a button when they aren’t SUPPOSED to be is… yeah.

Only within the bounds of how the game allows you to remap keys. You're basically just setting up macros. If your game's settings don't allow you to set up macros or remap keys a certain way, then G-Assist is of no use. It's just a convenience versus manually remapping.

 

[Onasi]

Only within the bounds of how the game allows you to remap keys. You're basically just setting up macros. If your game's settings don't allow you to set up macros or remap keys a certain way, then G-Assist is of no use. It's just a convenience versus manually remapping.

Okay then, no qualms there. You, uh, mind to add that to the article itself since that’s actually a significant difference?

 

[Fatalfury]

might as well let "AI play the game" and human "Assist the AI" instead when it's stuck somewhere.

 

[Vayra86]

'AI' please tell me what my FPS and power usage is... (or I can just hit my desired shortcut and show all stats on screen)

Darwin disagrees with this. He really does, I asked him.

 

[Chomiq]

Jesus, just what we needed, freaking Clippy to help us play the game.

 

[W1zzard]

'AI' please tell me what my FPS and power usage is... (or I can just hit my desired shortcut and show all stats on screen)

Darwin disagrees with this. He really does, I asked him.

Isn't your underlying, and harder question "how can I improve my FPS?".. and it does have pretty decent answers to that.

Check the last screenshot, the query was "lower power consumption, but make sure my FPS don't drop much below 60"

 

[Vayra86]

Isn't your underlying, and harder question "how can I improve my FPS?".. and it does have pretty decent answers to that.

Check the last screenshot, the query was "lower power consumption, but make sure my FPS don't drop much below 60"

Has it answered that... or has it done it for you, effectively taking 'work' out your hands.

And in that, has it not also taken knowledge out of your hands? Look, I can do this now. I think we're transitioning into an era where the skillset of the average human is outsourced. This is not a positive development if you ask me. We were already on that path, I recognize that too, even prior to AI. But I feel we're crossing the line here, and already reduced skillsets are painful for people. The gap between people of varying intelligence is also increasing: you either 'get it' or you never will. This also applies to the work people do: its quickly evolving into a 'have/have not' situation. The have nots are the ones that rely on prompts. The haves are the ones that control the machines.

On the bottom line, there is an increasing group in societies 'that just don't get it anymore', and AI isn't the fix for that, its part of the problem.

But that's thinking big.
Thinking smaller about this specific example...

Is this really such magic, or could we also just have a slider in the overclocking app that says 'maximum power usage' ? Because the algorithm is there already, we didn't need AI for it. Games already default you to the graphics settings that fit a desired FPS target, too...

 

[W1zzard]

Has it answered that... or has it done it for you, effectively taking 'work' out your hands.

It answered, and offered to do it, and done it for me, I had choices all the time

Is this really such magic, or could we also just have a slider in the overclocking app that says 'maximum power usage' ? Because the algorithm is there already, we didn't need AI for it.

You are absolutely right. The current OC software tooling that I'm aware of will give you sliders that control one variable, not solve problems by changing multiple variables.

 

[Onasi]

It answered, and offered to do it, and done it for me, I had choices all the time

Does it apply it on the fly with no noticeable delay? Is the change gradual or does it apply all the power tweaks immediately once the request by user is made?

 

[Assimilator]

I can actually see this being quite useful for RPGs. For example, you get to settlement X and ask the LLM "is there an important NPC that I should speak to while I'm here?", or "are there any quests I could start?", or "are there any particular hazards I should be aware of?", or "are there any points of interest near this location?" Especially when you are just starting out in a game and are a little overwhelmed, such information could be extremely helpful.

 

[Vayra86]

I can actually see this being quite useful for RPGs. For example, you get to settlement X and ask the LLM "is there an important NPC that I should speak to while I'm here?", or "are there any quests I could start?", or "are there any particular hazards I should be aware of?", or "are there any points of interest near this location?" Especially when you are just starting out in a game and are a little overwhelmed, such information could be extremely helpful.

Yeah god forbid you have to actually search for things yourself and actually spend time to figure it out.

Its very efficient, isn't it, getting the info you need as you go. But is entertainment about efficiency? Perhaps to a segment of the market.

 

[Chomiq]

I can actually see this being quite useful for RPGs. For example, you get to settlement X and ask the LLM "is there an important NPC that I should speak to while I'm here?", or "are there any quests I could start?", or "are there any particular hazards I should be aware of?", or "are there any points of interest near this location?" Especially when you are just starting out in a game and are a little overwhelmed, such information could be extremely helpful.

At what point playing the game becomes playing the "feed AI prompts" game? What next, we'll let the AI control the game because we don't want to bothered? Actually, that sort of makes sense for people with disabilities.

 

[L'Eliminateur]

honestly i don't get the negativity about this.
You talk about "cheating", but you do realize the time it takes to write the question and get the answer?, this will not affect any kind of "eSport"-trash or any PvP for that matter.

I see this as something i want yesterday, i can't even tell you how many times i've had to alt-tab a game to find some some walk-through due to poor game design and have to constantly tab in tab out to get the info i need to accomplish certain silly task/objective i don't care about in the game. With this i can get the AI to assist me INSIDE the game.

Think of stupid gather-unlock designs where you have to spend hours hunting "keys"/"pieces" in a maze-level(or several) only to get to the door and having the game tell you you still miss some and you have absolutely NO idea. Or dumb puzzles where the hint was some obscure NPC conversation or item half the world away ago that you saw 2 months ago with any luck or presented in some cryptic way(pathfinder WOTR is egregious, it might be months between my plays of that game).

Imagine loading a savegame you haven't touched in months/years and have absolutely not the faintest clue wtf you had to do and ask the chatbot for assistance on prioritizing the quests/items based on your map position

So yeah, i'm 200% hyped for this, will enrich my single player and coop gaming to absolute new levels of practicity

 

[Vayra86]

Of course, the bottom line is that you don't have to use it, if you don't want to. Good points above here, too.

 

[Assimilator]

Of course, the bottom line is that you don't have to use it, if you don't want to. Good points above here, too.

Exactly. As above, I might use this feature when I'm starting out in a game to help me get my bearings, but once I am comfortable I would turn it off.

I can also see it being useful in MMORPGs with timed events. "Hey LLM, what timed events are on now and where should I go to take part in them?"

 

[L'Eliminateur]

At what point playing the game becomes playing the "feed AI prompts" game? What next, we'll let the AI control the game because we don't want to bothered? Actually, that sort of makes sense for people with disabilities.

This speaks more about bad game design than player "laziness", you should be asking "why did they design it in a way that makes people want to autopilot this part?".

Gaming should be enjoyable, even more if you're a working adult and the gaming time is a precious commodity, as subjective as "enjoyment" is for each player and game type there are some universal responses for each game type/genre(for example: you might expect "rogue-like/lite" games to not feature on demand savegame, sould-like games to have punishing difficulty, survival games to feature lots of crafting) so i see this as a very welcome extra for those cases where a game turns tedious for you or you become stuck, or don't have time to pour through 20 forum threads to learn what the current "meta" is.

 

[redeye]

remember the 2001:space Odyssey quote?… (and apologies to the film ) (original quote from https://www.kpbs.org/news/news/scie...door-hal-heres-how-ai-became-a-movie-villain#)
because it was the first hit in google…)

coming soon to g.assist…

“Open the pod bay door, g.assist" became one of the most quoted game situations of the decade when g.assist responded, "I'm sorry, Dave, I'm afraid I can't do that. This mission is too important for me to allow you to jeopardize it."

get it? “ g spot assist” …

 

[W1zzard]

Does it apply it on the fly with no noticeable delay? Is the change gradual or does it apply all the power tweaks immediately once the request by user is made?

There is no delay, stutter, lag or anything. Processing the prompt does take a second or so, which is probably due to the LLM, not sure to some settings change

 

[evernessince]

I can actually see this being quite useful for RPGs. For example, you get to settlement X and ask the LLM "is there an important NPC that I should speak to while I'm here?", or "are there any quests I could start?", or "are there any particular hazards I should be aware of?", or "are there any points of interest near this location?" Especially when you are just starting out in a game and are a little overwhelmed, such information could be extremely helpful.

Fair enough if you use that when you are stuck and have exhausted your options or when the game is not intuitive enough to give you that information naturally but using it otherwise is just robbing yourself of the experience of playing the game.

 

[Vayra86]

Exactly. As above, I might use this feature when I'm starting out in a game to help me get my bearings, but once I am comfortable I would turn it off.

I can also see it being useful in MMORPGs with timed events. "Hey LLM, what timed events are on now and where should I go to take part in them?"

Warframe would love this... might even consider playing it again. God thats some mess of possible things to do.

Fair enough if you use that when you are stuck and have exhausted your options or when the game is not intuitive enough to give you that information naturally but using it otherwise is just robbing yourself of the experience of playing the game.

Yeah but cheats do that too. You experience that quickly and then you can make a very informed choice if you want to keep doing it.

 

[JIWIL]

Kinda get the feeling that AI botting is going to explode in the next few years, making most MMOs and even co-op/competitive stuff pretty irrelevant. It's been happening slowly over the past decade or so, but slope is getting really slippery at this point. Monitors with AI in them that highlight opponents, AI driven mice that snap-on headshots.. a bot that will run all your bots killing them nameds for sweet loot, etc.

 

[Random_User]

Only within the bounds of how the game allows you to remap keys. You're basically just setting up macros. If your game's settings don't allow you to set up macros or remap keys a certain way, then G-Assist is of no use. It's just a convenience versus manually remapping.

AFAIK, most multiplayer, at least FPS games, had marcros considered cheat, anyway, for a very long time. At least from my experience.