I experimented with this exact same approach earlier this year.
It's barely sufficient, because, bluntly, most apps just aren't wired up right.
So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out.
Will try this app, to see if it improved on my own approach, but man, the hope levels are low.
I hope in near future _that_ layer of abstraction -- looking at a fairly standard application window with minor UI variations and reasoning about what area / labels within the UI mean what (possibly paired with app documentation) -- could probably become a light-weight fine-tuned vision model it itself that can run fully locally.
I also saw that HeyClicky started doing something similar but end up removing from the product.
> It writes plain markdown
Where are the formatting decisions coming from?
It's barely sufficient, because, bluntly, most apps just aren't wired up right.
So you end up having to hand code a lot of specific profiles for specific apps to make this work well, and even then, you don't quite get the right level of detail to make it work out.
Will try this app, to see if it improved on my own approach, but man, the hope levels are low.