[ad_1]
Apple researchers have published yet another paper on artificial intelligence (AI) models, and this time the focus is on understanding and navigating through smartphone user interfaces (UI). The yet-to-be peer-reviewed research paper highlights a large language model (LLM) dubbed Ferret UI, which can go beyond traditional computer vision and understand complex smartphone screens. Notably, this is not the first paper on AI published by the research division of the tech giant. It has already published a paper on multimodal LLMs (MLLMs) and another on on-device AI models.
The pre-print version of the research paper has been published on arXiv, an open-access online repository of scholarly papers. The paper is titled “Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs” and focuses on expanding the use case of MLLMs. It highlights that most language models with multimodal capabilities cannot understand beyond natural images and are functionality “restricted”. It also states the need for AI models to understand complex and dynamic interfaces such as those on a smartphone.
As per the paper, Ferret UI is “designed to execute precise referring and grounding tasks specific to UI screens, while adeptly interpreting and acting upon open-ended language instructions.” In simple terms, the vision language model can not only process a smartphone screen with multiple elements representing different information but it can also tell a user about them when prompted with a query.
How Ferret UI processes information on a screen
Photo Credit: Apple
Based on an image shared in the paper, the model can understand and classify widgets and recognise icons. It can also answer questions such as “Where is the launch icon”, and “How do I open the Reminders app”. This shows that the AI is not only capable of explaining the screen it sees, but can also navigate to different parts of an iPhone based on a prompt.
To train Ferret UI, the Apple researchers created data of varying complexities themselves. This helped the model in learning basic tasks and understanding single-step processes. “For advanced tasks, we use GPT-4 [40] to generate data, including detailed description, conversation perception, conversation interaction, and function inference. These advanced tasks prepare the model to engage in more nuanced discussions about visual components, formulate action plans with specific goals in mind, and interpret the general purpose of a screen,” the paper explained.
The paper is promising, and if it passes the peer-review stage, Apple might be able to utilise this capability to add powerful tools to the iPhone that can perform complex UI navigation tasks with simple text or verbal prompts. This capability appears to be ideal for Siri.
[ad_2]
Source link
medication from mexico pharmacy: mexican online pharmacy – pharmacies in mexico that ship to usa
canadian pharmacy no scripts [url=http://canadapharmast.com/#]canadianpharmacyworld[/url] vipps approved canadian online pharmacy
canada drugstore pharmacy rx: canadian pharmacies compare – canada pharmacy reviews
canadian discount pharmacy: best canadian online pharmacy – vipps canadian pharmacy
pharmacies in mexico that ship to usa [url=http://foruspharma.com/#]mexico drug stores pharmacies[/url] mexico pharmacy
top 10 online pharmacy in india: buy prescription drugs from india – cheapest online pharmacy india
canadian pharmacy in canada [url=http://canadapharmast.com/#]safe canadian pharmacy[/url] canadian pharmacy no scripts
mexican rx online: best online pharmacies in mexico – mexico pharmacies prescription drugs
indianpharmacy com [url=https://indiapharmast.com/#]best india pharmacy[/url] indian pharmacy online
http://amoxildelivery.pro/# amoxacillian without a percription
buy cipro online canada [url=http://ciprodelivery.pro/#]ciprofloxacin[/url] cipro for sale
https://doxycyclinedelivery.pro/# buying doxycycline online
doxycycline 40 mg capsules [url=https://doxycyclinedelivery.pro/#]doxycycline 100mg online uk[/url] doxycycline cap price
https://ciprodelivery.pro/# ciprofloxacin order online
cipro online no prescription in the usa [url=http://ciprodelivery.pro/#]buy cipro online without prescription[/url] cipro 500mg best prices
https://paxloviddelivery.pro/# п»їpaxlovid
40mg doxycycline online [url=https://doxycyclinedelivery.pro/#]doxycycline 120mg[/url] doxycycline hydrochloride
https://ciprodelivery.pro/# purchase cipro
Paxlovid buy online [url=https://paxloviddelivery.pro/#]Paxlovid buy online[/url] п»їpaxlovid
http://amoxildelivery.pro/# amoxicillin over counter
can you buy amoxicillin over the counter [url=https://amoxildelivery.pro/#]price for amoxicillin 875 mg[/url] amoxicillin 500mg no prescription
http://amoxildelivery.pro/# amoxicillin 250 mg capsule
buy generic ciprofloxacin [url=https://ciprodelivery.pro/#]cipro for sale[/url] cipro pharmacy
buying from online mexican pharmacy [url=https://mexicandeliverypharma.com/#]buying prescription drugs in mexico online[/url] п»їbest mexican online pharmacies
medication from mexico pharmacy [url=https://mexicandeliverypharma.online/#]medication from mexico pharmacy[/url] reputable mexican pharmacies online
medication from mexico pharmacy [url=http://mexicandeliverypharma.com/#]mexican border pharmacies shipping to usa[/url] mexico pharmacies prescription drugs
buying prescription drugs in mexico: buying prescription drugs in mexico – reputable mexican pharmacies online
mexican rx online [url=http://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] mexican pharmaceuticals online
mexican pharmaceuticals online: purple pharmacy mexico price list – medicine in mexico pharmacies
purple pharmacy mexico price list [url=http://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] buying from online mexican pharmacy
mexican rx online [url=https://mexicandeliverypharma.online/#]mexico drug stores pharmacies[/url] buying prescription drugs in mexico online
buying from online mexican pharmacy [url=https://mexicandeliverypharma.com/#]medication from mexico pharmacy[/url] best online pharmacies in mexico
mexican mail order pharmacies [url=https://mexicandeliverypharma.com/#]mexican rx online[/url] mexico pharmacy
mexico pharmacy [url=https://mexicandeliverypharma.com/#]mexico drug stores pharmacies[/url] reputable mexican pharmacies online
mexican border pharmacies shipping to usa [url=https://mexicandeliverypharma.online/#]mexican drugstore online[/url] mexican online pharmacies prescription drugs
mexican pharmaceuticals online [url=https://mexicandeliverypharma.online/#]purple pharmacy mexico price list[/url] п»їbest mexican online pharmacies
п»їbest mexican online pharmacies [url=http://mexicandeliverypharma.com/#]pharmacies in mexico that ship to usa[/url] mexican border pharmacies shipping to usa
comprare farmaci online con ricetta: Tadalafil generico migliore – Farmacie on line spedizione gratuita
farmaci senza ricetta elenco: Avanafil prezzo – farmaci senza ricetta elenco
farmacia online: kamagra gel – Farmacie on line spedizione gratuita
farmacia online: Avanafil prezzo – acquisto farmaci con ricetta
farmacia online senza ricetta: Cialis generico controindicazioni – Farmacia online piГ№ conveniente
mail order cialis: indian cialis generic – order cialis canada
http://sildenafil.llc/# generic viagra without a doctor prescription
viagra generic: Viagra online price – cost of viagra
http://tadalafil.auction/# where to get the best price on cialis
generic viagra overnight: Cheap generic Viagra – cheap viagra
https://sildenafil.llc/# viagra prices
cheap viagra [url=https://sildenafil.llc/#]Cheap Viagra 100mg[/url] viagra online
generic cialis with dapoxetine 80mg: Buy Cialis online – cialis on line pharm
https://tadalafil.auction/# canadian pharmacy no prescription cialis
how to get ed pills: Best ED meds online – buy erectile dysfunction medication
https://indiapharmacy.shop/# indianpharmacy com
cheapest ed meds
https://edpillpharmacy.store/# buy erectile dysfunction medication
reputable mexican pharmacies online: Certified Mexican pharmacy – buying prescription drugs in mexico online
http://edpillpharmacy.store/# ed rx online
cheap ed meds online
https://mexicopharmacy.win/# mexican border pharmacies shipping to usa
http://edpillpharmacy.store/# cheapest online ed meds
erectile dysfunction medicine online
online ed medicine: Cheap ED pills online – ed pills for sale
top rated ed pills: ED meds online with insurance – best online ed treatment
https://mexicopharmacy.win/# mexican border pharmacies shipping to usa
http://edpillpharmacy.store/# cheap ed meds online
http://edpillpharmacy.store/# online ed medication
http://indiapharmacy.shop/# Online medicine home delivery
http://mexicopharmacy.win/# buying from online mexican pharmacy
https://edpillpharmacy.store/# erectile dysfunction medicine online
purchase cytotec https://cytotec.pro/# buy cytotec
lasix generic name
https://lipitor.guru/# lipitor 80 mg tablet
cytotec buy online usa https://tamoxifen.bid/# generic tamoxifen
lasix online
how much is lipitor discount [url=https://lipitor.guru/#]cheapest ace inhibitor[/url] lipitor 10 mg
https://furosemide.win/# lasix generic name
https://cytotec.pro/# cytotec online
cytotec abortion pill http://furosemide.win/# furosemide
buy lasix online
lisinopril 20mg pill [url=https://lisinopril.guru/#]cheap lisinopril[/url] 40 mg lisinopril for sale
cytotec abortion pill: cytotec best price – buy cytotec online fast delivery
https://furosemide.win/# lasix tablet
lisinopril price in canada: Lisinopril refill online – buy lisinopril online no prescription india
п»їcytotec pills online https://tamoxifen.bid/# aromatase inhibitor tamoxifen
furosemida
tamoxifen bone density: Purchase Nolvadex Online – common side effects of tamoxifen
http://lipitor.guru/# lipitor prices australia
buy cytotec online fast delivery http://cytotec.pro/# cytotec buy online usa
lasix 100 mg
furosemide: buy furosemide – buy lasix online
http://tamoxifen.bid/# tamoxifen menopause
Misoprostol 200 mg buy online https://furosemide.win/# furosemide 100mg
furosemida 40 mg
https://easyrxindia.shop/# Online medicine order
http://mexstarpharma.com/# mexican pharmaceuticals online
https://easyrxindia.shop/# indian pharmacy
http://mexstarpharma.com/# mexican online pharmacies prescription drugs
http://mexstarpharma.com/# medicine in mexico pharmacies
http://easyrxindia.com/# indianpharmacy com
https://mexstarpharma.com/# buying from online mexican pharmacy
https://easyrxcanada.com/# precription drugs from canada
bonus veren siteler: bahis siteleri – deneme bonusu
https://denemebonusuverensiteler.win/# bahis siteleri
en iyi slot siteleri 2024: slot oyun siteleri – en cok kazandiran slot siteleri
https://slotsiteleri.bid/# en cok kazandiran slot siteleri
deneme veren slot siteleri: en guvenilir slot siteleri – bonus veren slot siteleri
canl? slot siteleri: yasal slot siteleri – guvenilir slot siteleri
https://slotsiteleri.bid/# yasal slot siteleri
ван вин: 1вин – 1win официальный сайт
https://pin-up.diy/# пин ап вход
vavada casino [url=http://vavada.auction/#]vavada online casino[/url] вавада
пин ап казино: пин ап казино – пинап казино
http://1win.directory/# 1win вход
зеркало 1хбет: 1хбет официальный сайт – 1хбет официальный сайт
пин ап казино вход: пинап казино – пин ап казино вход
http://1win.directory/# 1вин
vavada online casino: vavada casino – вавада рабочее зеркало
1xbet зеркало: 1xbet официальный сайт мобильная версия – 1xbet зеркало рабочее на сегодня