Nearly a decade ago, Google introduced a feature called Now on Tap in Android Marshmallow. By tapping and holding the home button, Google would provide relevant contextual information based on what was on the screen. For example, discussing a movie with a friend over text could prompt Now on Tap to display details about the film without leaving the messaging app. Similarly, viewing a restaurant on Yelp could trigger the phone to show OpenTable recommendations with a simple tap.
At that time, fresh out of college, I found these enhancements thrilling and almost magical. The ability to recognize on-screen content and anticipate user actions felt ahead of its time. Now on Tap quickly evolved into Google Assistant, which was impressive in its own right but lacked the same charm.
Fast forward to today at Google’s I/O developer conference in Mountain View, California, where the latest Android features evoke memories of the original Now on Tap. These features leverage contextual information to simplify phone usage, now powered by a decade’s worth of advancements in large language models.
Dave Burke, vice president of engineering on Android, remarked, “I think what’s exciting is we now have the technology to build really exciting assistants. We now possess the capability to develop a computer system that comprehends what it sees, a feat we struggled with in the past.” I had the opportunity to discuss these new developments with Burke and Sameer Samat, president of the Android ecosystem at Google, shedding light on Android’s future and the introduction of the new AI assistant Gemini.
Circle to Search … Your Homework
The upgraded Circle to Search in action.
Courtesy of Google
The introduction of Circle to Search marks Google’s fresh approach to mobile search. Similar to Now on Tap, Circle to Search encourages a more interactive search experience than traditional typing. By circling the desired content on the screen, users can initiate searches in a visually engaging manner. Burke emphasized, “It’s a very visceral, fun, and modern way to search. It also appeals to a younger audience due to its interactive nature.”
Samat mentioned positive consumer feedback for Circle to Search, particularly highlighting its new feature tailored towards students. The updated Circle to Search now offers step-by-step instructions for solving physics and math problems. Users can circle the problems within their syllabus app, and Google will provide detailed solutions without the need to switch apps.
Samat emphasized that Gemini isn’t merely giving answers but guiding students through problem-solving processes. In the future, Circle to Search will tackle more complex challenges like diagrams and graphs, thanks to Google’s specialized LearnLM models for education.
Gemini Gets More Contextual on Android
Gemini, Google’s AI assistant, is gradually surpassing Google Assistant in functionality. On most Android phones today, users have the option to replace Google Assistant with Gemini. When asked about the fate of Assistant, Samat clarified, “Gemini is an opt-in experience on the phone. It continues to evolve and advance, offering consumers a choice to try out this new AI-powered assistant. The positive feedback we’ve received indicates that users are embracing this option.”
