UseDesktop - An infrastructure layer for training deskop agents

@vouchy That's a good question :) What made me make an agent that is between deterministic and non-deterministic nature.

Well, the answer is because "Everybody has their own preference and expected process and result"

I will try to explain this with an easy example.

I have been building a desktop agent that works/acts on your PC for about 1 year.

This was one of the first MVP video I made (April, 2025)

At first, the actions were completely based on the prompts and LLM's non-deterministic nature.

and The problem with that was it started doing things that wasn't what i liked of or thinking of.

For example, when i asked the agent to log in to certain webpage, what you expect a normal human to do is write the id and password in the forms and click Login button. However, All the agents tried to just put the ID and password in the url as query string and login. It worked sometimes but most of the time, such unexpected behavior led to errors and the actions they do are not constant at all which the unexpected behaviors often lead to failures ,thus, no actual use cases could be made except some fake hypes.

Secondly, everybody has their own preference. What I mean by that is that I personally prefer to open APP using command + space in my mac but for some people, they would prefer to open app using double click.

Now what would LLM do in such cases? you never know and that is the problem.

Employer wants their employees to work in specific way the employer thinks of, you most of the time don't want people that works very differently and randomly. Same goes for Human - Agent relationship, you prompt things hoping the agent to work in specific way and increase productivity or make money not that agent does whatever he wants to do :)

That was just a basic example but in a complex system where things should be in order/exactly done, this often leads to a problem as LLMs are random.

That is the reason why I started taking it more serious approach how agent should tackle such problems.

I first went with browser use, but this did not work also (I will make a seperate story about it later why browser use shouldn't be the go to

I tried with cli use but this also did not work too.

So now i completely went with GUI based and it is

• much faster

• less cost

• and more accurate

and I believe to make an actual agent that can perform (i.e real use cases) at human level, making agent works in confined and restricted env like browser or cli would not work. The agent should work just like human does imo.

If you have any other questions, I will be happy to answer :)