Apple Researchers Are Building AI Model Called ‘Ferret UI’ That Can Navigate Through iOS – Life Changer

Apple Researchers Are Building AI Model Called ‘Ferret UI’ That Can Navigate Through iOS

248

[ad_1]

Apple researchers have published yet another paper on artificial intelligence (AI) models, and this time the focus is on understanding and navigating through smartphone user interfaces (UI). The yet-to-be peer-reviewed research paper highlights a large language model (LLM) dubbed Ferret UI, which can go beyond traditional computer vision and understand complex smartphone screens. Notably, this is not the first paper on AI published by the research division of the tech giant. It has already published a paper on multimodal LLMs (MLLMs) and another on on-device AI models.

The pre-print version of the research paper has been published on arXiv, an open-access online repository of scholarly papers. The paper is titled “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs” and focuses on expanding the use case of MLLMs. It highlights that most language models with multimodal capabilities cannot understand beyond natural images and are functionality “restricted”. It also states the need for AI models to understand complex and dynamic interfaces such as those on a smartphone.

As per the paper, Ferret UI is “designed to execute precise referring and grounding tasks specific to UI screens, while adeptly interpreting and acting upon open-ended language instructions.” In simple terms, the vision language model can not only process a smartphone screen with multiple elements representing different information but it can also tell a user about them when prompted with a query.

How Ferret UI processes information on a screen
Photo Credit: Apple

 

Based on an image shared in the paper, the model can understand and classify widgets and recognise icons. It can also answer questions such as “Where is the launch icon”, and “How do I open the Reminders app”. This shows that the AI is not only capable of explaining the screen it sees, but can also navigate to different parts of an iPhone based on a prompt.

To train Ferret UI, the Apple researchers created data of varying complexities themselves. This helped the model in learning basic tasks and understanding single-step processes. “For advanced tasks, we use GPT-4 [40] to generate data, including detailed description, conversation perception, conversation interaction, and function inference. These advanced tasks prepare the model to engage in more nuanced discussions about visual components, formulate action plans with specific goals in mind, and interpret the general purpose of a screen,” the paper explained.

The paper is promising, and if it passes the peer-review stage, Apple might be able to utilise this capability to add powerful tools to the iPhone that can perform complex UI navigation tasks with simple text or verbal prompts. This capability appears to be ideal for Siri.

Affiliate links may be automatically generated – see our ethics statement for details.

[ad_2]

Source link

Loading

Post Views: 690

248 thoughts on “Apple Researchers Are Building AI Model Called ‘Ferret UI’ That Can Navigate Through iOS”

  2. canadian pharmacy no scripts [url=http://canadapharmast.com/#]canadianpharmacyworld[/url] vipps approved canadian online pharmacy

    Reply

  5. pharmacies in mexico that ship to usa [url=http://foruspharma.com/#]mexico drug stores pharmacies[/url] mexico pharmacy

    Reply

  7. canadian pharmacy in canada [url=http://canadapharmast.com/#]safe canadian pharmacy[/url] canadian pharmacy no scripts

    Reply

  9. indianpharmacy com [url=https://indiapharmast.com/#]best india pharmacy[/url] indian pharmacy online

    Reply

  17. buying from online mexican pharmacy [url=https://mexicandeliverypharma.com/#]buying prescription drugs in mexico online[/url] п»їbest mexican online pharmacies

    Reply

  18. medication from mexico pharmacy [url=https://mexicandeliverypharma.online/#]medication from mexico pharmacy[/url] reputable mexican pharmacies online

    Reply

  19. medication from mexico pharmacy [url=http://mexicandeliverypharma.com/#]mexican border pharmacies shipping to usa[/url] mexico pharmacies prescription drugs

    Reply

  21. mexican rx online [url=http://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] mexican pharmaceuticals online

    Reply

  23. purple pharmacy mexico price list [url=http://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] buying from online mexican pharmacy

    Reply

  24. mexican rx online [url=https://mexicandeliverypharma.online/#]mexico drug stores pharmacies[/url] buying prescription drugs in mexico online

    Reply

  25. buying from online mexican pharmacy [url=https://mexicandeliverypharma.com/#]medication from mexico pharmacy[/url] best online pharmacies in mexico

    Reply

  26. mexican mail order pharmacies [url=https://mexicandeliverypharma.com/#]mexican rx online[/url] mexico pharmacy

    Reply

  27. mexico pharmacy [url=https://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] reputable mexican pharmacies online

    Reply

  28. mexican border pharmacies shipping to usa [url=https://mexicandeliverypharma.online/#]mexican drugstore online[/url] mexican online pharmacies prescription drugs

    Reply

  29. mexican pharmaceuticals online [url=https://mexicandeliverypharma.online/#]purple pharmacy mexico price list[/url] п»їbest mexican online pharmacies

    Reply

  30. п»їbest mexican online pharmacies [url=http://mexicandeliverypharma.com/#]pharmacies in mexico that ship to usa[/url] mexican border pharmacies shipping to usa

    Reply

  117. online pharmacy india [url=http://indianpharmacy.company/#]top online pharmacy india[/url] buy medicines online in india

    Reply

  123. pharmacies in mexico that ship to usa [url=https://mexicopharmacy.cheap/#]buying prescription drugs in mexico[/url] mexican border pharmacies shipping to usa

    Reply

  127. reputable mexican pharmacies online [url=https://mexicopharmacy.cheap/#]mexican mail order pharmacies[/url] buying from online mexican pharmacy

    Reply

  137. buying from online mexican pharmacy [url=http://mexicopharmacy.cheap/#]buying prescription drugs in mexico[/url] п»їbest mexican online pharmacies

    Reply

  140. lamisil online pharmacy [url=http://pharmbig24.com/#]rx pharmacy coupons reviews[/url] australian pharmacy domperidone

    Reply

  150. buying prescription drugs in mexico online [url=https://mexicopharmacy.cheap/#]purple pharmacy mexico price list[/url] pharmacies in mexico that ship to usa

    Reply

  171. casibom guncel giris: casibom – casibom
    gates of olympus oyna [url=http://gatesofolympusoyna.online/#]gates of olympus demo turkce oyna[/url] gates of olympus demo turkce oyna

    Reply

  176. comprar sildenafilo cinfa 100 mg espaГ±a [url=http://sildenafilo.men/#]viagra precio[/url] comprar viagra en espaГ±a

    Reply

  179. comprar viagra en espaГ±a envio urgente [url=http://sildenafilo.men/#]sildenafilo precio[/url] sildenafilo 50 mg precio sin receta

    Reply

  186. farmacia online piГ№ conveniente [url=http://farmaciait.men/#]Farmacia online piu conveniente[/url] top farmacia online

    Reply

  191. migliori farmacie online 2024 [url=https://brufen.pro/#]BRUFEN 600 bustine prezzo[/url] Farmacie on line spedizione gratuita

    Reply

  199. farmacia online senza ricetta [url=http://farmaciait.men/#]farmacia online migliore[/url] comprare farmaci online all’estero

    Reply

  204. comprare farmaci online con ricetta [url=https://tadalafilit.com/#]Farmacie che vendono Cialis senza ricetta[/url] farmaci senza ricetta elenco

    Reply

  213. Viagra Pfizer sans ordonnance [url=https://vgrsansordonnance.com/#]Viagra generique en pharmacie[/url] Prix du Viagra 100mg en France

    Reply

  214. Achat mГ©dicament en ligne fiable [url=https://clssansordonnance.icu/#]Cialis sans ordonnance 24h[/url] pharmacie en ligne sans ordonnance

    Reply

  215. pharmacie en ligne france fiable [url=http://pharmaciepascher.pro/#]Medicaments en ligne livres en 24h[/url] Pharmacie Internationale en ligne

    Reply

  219. My developer is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the costs.
    But he’s tryiong none the less. I’ve been using Movable-type on numerous websites for
    about a year and am concerned about switching to another platform.
    I have heard good things about blogengine.net. Is there a way I can transfer all my wordpress posts into it?

    Any kind of help would be greatly appreciated!

    Reply

  222. After I originally commented I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on every time a comment is added I get 4 emails with the
    same comment. There has to be an easy method you
    are able to remove me from that service?
    Thank you!

    Reply

  224. I was recommended this website by my cousin. I am not
    sure whether this post is written by him as no one else know such detailed about my problem.
    You are amazing! Thanks!

    Reply

  225. Hi there just wanted to give you a brief heads up and let you know a few of the pictures aren’t
    loading correctly. I’m not sure why but I think its a linking issue.
    I’ve tried it in two different internet browsers and both show
    the same results.

    Reply

  226. It’s truly very difficult in this active life to listen news on Television, therefore I just use world wide web for that reason, and obtain the newest information.

    Reply

  227. Definitely believe that that you stated. Your favorite reason seemed to be on the internet the
    simplest thing to consider of. I say to you, I certainly get irked whilst people think about concerns that they plainly do not understand about.
    You controlled to hit the nail upon the highest and also defined out the whole thing without
    having side-effects , other folks can take a signal.
    Will likely be again to get more. Thanks

    Reply

  229. Hi! I could have sworn I’ve visited this website before but after going through some of the
    articles I realized it’s new to me. Anyways, I’m definitely pleased I discovered it
    and I’ll be book-marking it and checking back regularly!

    Reply

  230. I have been surfing online more than 2 hours today,
    yet I never found any interesting article like yours. It is pretty worth enough for me.
    Personally, if all site owners and bloggers made good content
    as you did, the net will be much more useful than ever before.

    Reply

  231. Does your blog have a contact page? I’m having trouble locating
    it but, I’d like to shoot you an e-mail. I’ve got some recommendations for your blog you might be interested in hearing.

    Either way, great blog and I look forward to seeing it develop over time.

    Reply

  233. Hey! I know this is kinda off topic but I was wondering which blog platform are you using for
    this site? I’m getting sick and tired of WordPress because I’ve had
    issues with hackers and I’m looking at options for another platform.
    I would be fantastic if you could point me in the direction of a good platform.

    Reply

  234. Wow, amazing blog layout! How long have you been blogging
    for? you make blogging look easy. The overall look of your
    website is great, as well as the content!

    Reply

  235. Tremendous issues here. I am very satisfied to peer your post.
    Thank you so much and I’m taking a look ahead to touch you.

    Will you please drop me a mail?

    Reply

  236. I’m amazed, I have to admit. Seldom do I come across a blog that’s equally educative and
    amusing, and without a doubt, you’ve hit the
    nail on the head. The problem is something that too few men and
    women are speaking intelligently about. I’m
    very happy that I stumbled across this during
    my hunt for something regarding this.

    Reply

  237. Hi there would you mind stating which blog platform you’re
    working with? I’m going to start my own blog soon but I’m having a difficult time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design seems different then most blogs and
    I’m looking for something unique. P.S Apologies for being off-topic but
    I had to ask!

    Reply

  239. Great article! That is the type of info that should be shared across the internet.
    Shame on the search engines for no longer positioning this post upper!
    Come on over and seek advice from my website . Thank you =)

    Reply

  240. Nice blog here! Also your web site loads up fast! What web host are you using?

    Can I get your affiliate link to your host? I
    wish my site loaded up as quickly as yours lol

    Reply

  242. Hey there would you mind stating which blog platform you’re
    using? I’m planning to start my own blog in the near future but I’m having a
    tough time deciding between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design seems
    different then most blogs and I’m looking for something completely unique.
    P.S Apologies for getting off-topic but I had to ask!

    Reply

  243. Everyone loves what you guys are up too. This kind of clever work and exposure!

    Keep up the wonderful works guys I’ve incorporated you guys to my blogroll.

    Reply

  244. Greetings from California! I’m bored at work so I decided to browse your website on my iphone during
    lunch break. I love the knowledge you provide here
    and can’t wait to take a look when I get home. I’m amazed at
    how quick your blog loaded on my phone .. I’m not even using WIFI, just 3G ..
    Anyways, wonderful blog!

    Reply

  245. Hey there! I’m at work browsing your blog from
    my new iphone 3gs! Just wanted to say I love reading through
    your blog and look forward to all your posts!
    Carry on the superb work!

    Reply

  248. I loved as much as you will receive carried out right here.
    The sketch is attractive, your authored material stylish.
    nonetheless, you command get bought an nervousness over that you wish be delivering the following.
    unwell unquestionably come further formerly again as exactly the same
    nearly very often inside case you shield this increase.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Articles

From Omnichannel POS Systems to AI-Powered Chatbots, Technology is the Future of the Restaurant Industry |
From Omnichannel POS Systems to AI-Powered Chatbots, Technology is the Future of the Restaurant Industry |
[ad_1] The adoption of omnichannel POS platforms has transformed how restaurants operate. Instead of just…
Read More
Writers Guild of Canada Overwhelmingly Votes to Authorize Strike Over AI, Fair Pay
Writers Guild of Canada Overwhelmingly Votes to Authorize Strike Over AI, Fair Pay
[ad_1] THR Illustration / Adobe Stock Canadian writers appear ready to strike as they hold…
Read More
Congo questions Apple over knowledge of conflict minerals in its supply chain | WGN Radio 720
Congo questions Apple over knowledge of conflict minerals in its supply chain | WGN Radio 720
[ad_1] By GERALD IMRAY, Associated Press 42 mins ago FILE – A Congolese miner sifts…
Read More