The latest Ray-Bans currently offer some limited Meta AI capabilities, such as taking photos using voice. But now Meta is significantly expanding the scope of what its new bot can do: On Tuesday, the company announced that it’s testing a new “multimodal” AI feature that can recognize objects, hear requests, and sense requests as seen through smart glasses. From describing foods to offering style guidance, answer relevant questions about them.
The key to this “early access” experience is the outward-facing camera system in the glasses frames. Ray-Bans don’t pack the powerful processor chips found in the latest iPhones and Pixel smartphones, for example, so they can’t do onboard calculations or operations. In order for the glasses to respond, Meta AI must send requests and images to the company’s servers for processing. This may cause a delay of a few seconds, but the engineering team is actively working to shorten the delay.
More from WWD
When the user says a question or command out loud, the device captures the images the person sees and transmits them to the company’s servers; Thus, Meta AI can understand what the user is looking at and provide a relevant response. It’s a bit like stuffing Google Lens into a pair of Amazon Echo Frames.
In a demo for WWD, the glasses were pointed at a multicolored patterned upper and the bot was asked: “Hey Meta, look and tell me what this upper goes with.” Meta AI suggested dark pants to start the push. Studded black leather ankle boots were recommended to be paired with jeans. But it can’t specify a specific set of jeans or search for stores that sell those boots like Google Lens can. At least not yet.
“Sometimes you will hear this in an error message; When you try this in early access, you’ll hear, ‘I’m sorry, I can’t answer questions about the product, but I’m working on getting this feature soon.'” Anant Narayanan, Meta’s director of smart glasses engineering, told WWD. “To get this right, we have to specifically We know there is more work required. But for now, in early access, more general ‘what would go well with this’ type questions are being asked.”
According to Narayanan, the wait may not be long as the team aims to introduce product-focused capabilities next year. This shouldn’t surprise anyone. Meta AI is spreading across the company’s portfolio of devices and social media, and social commerce is on track to become a billion-dollar business this year. Statista predicts 2023 global sales on Instagram, Facebook and other platforms will be around $1.3 billion by the end of the year. This could increase further in 2024, thanks to Meta’s latest partnership with Amazon. The e-commerce giant is making deals to integrate its shopping platform more directly with social networks.
Meanwhile, there are various Meta AI queries and commands that testers should check. All initial trigger phrases start the same way: “Hey Meta, look and…” – like “Hey Meta, look and tell me what recipes I can make with these ingredients” or “Hey Meta, look and summarize this article for me.” .” Once activated, the bot remembers the context so it can understand shorter follow-up commands without having to repeat the “look” each time. It can also answer questions and offer language translations on the go when the user points to a specific word or phrase in a text-based document, restaurant menu, or WWD article.
In the demo, the bot identified food items, suggested recipes using specific ingredients, read restaurant menus to highlight spicy or vegetarian dishes, and even created a humorous description of artworks placed in the lobby upon request. Some commands worked better than others. In one example, Meta AI misidentified dragon fruit as a pomegranate, likely due to the bot’s ability to cling to context. When history was erased, the strangeness disappeared.
The user’s ability to delete the AI’s history and images is part of the company’s stated privacy push; This seems logical, considering the longstanding privacy criticisms and legal issues that have dogged Meta. This is also a priority in how AI is developed. Narayanan told WWD that the tool is trained with a mix of data either collected specifically by Meta or selected from its family of apps, but in the latter case, only when users consent to their data being used in this way.
Naturally, as a testing feature, the new Meta AI functionality isn’t perfect. But it’s still exciting for the developers behind the scenes. They’re eager to see how testers use the bot and parse feedback for improvements. This user input comes in a format very similar to Facebook, with a like or downvote rating system for each interaction, as recorded in the Meta View app.
Meta AI was first introduced at Meta Connect in September, and the bot has since spread to the Quest 3 headset, Instagram, WhatsApp and Messenger apps, as well as the latest Ray-Ban glasses. Meanwhile, Chief Executive Officer Mark Zuckerberg is giving the public a glimpse of what’s in store. In an Instagram post from October, “Zuck” marveled at his daughter’s newfound ability to braid her hair, posting the following video: “Finally learned to braid. Thank you Meta AI.”
Early access to this latest feature will be opened to testers selected from a pool of US users who register their interest online. Andrew “Boz” Bosworth, Meta’s chief technology officer, shared the news on Instagram and other social media on Tuesday: “We are testing multi-modal AI in beta on Ray-Ban Meta glasses through an optional early access program (US only). It’s early days, but I’m excited that this will enable Meta AI to become increasingly useful, especially in the eyewear form factor.” Meta is also rolling out the ability for Ray-Ban users in the US to request real-time information from Meta AI through searches powered in part by Bing .
Zuckerberg’s video Instagram post highlighted his style aspect. The CEO, probably wearing glasses, held up a brown shirt with multicolored stripes and said: “Hey Meta, look and tell me which pants to wear with this shirt.”
Meta AI responded: “From the image it looks like a striped shirt. A pair of dark wash jeans or solid color trousers will complement this shirt very well.”
The best of WWD