​We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take as input text instructions and visual egocentric observations and output… Read More »Large Language Models as Generalizable Policies for Embodied Tasks Apple Machine Learning Research

​In this paper, we introduce a novel approach to automatically assign entity labels to images from existing noisy image-text pairs. The approach employees a named entity recognition model to extract entities from text, and uses a CLIP model to select the right entities as the… Read More »MOFI: Learning Image Representation from Noisy Entity Annotated Images Apple Machine Learning Research

​[[{“value”:”This paper was accepted at the How Far Are We from AGI? workshop at ICLR 2024. Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated incredible strides on diverse vision language tasks. We dig into vision-based deductive reasoning, a more sophisticated but less explored realm,… Read More »How Far Are We from Intelligent Visual Deductive Reasoning? Apple Machine Learning Research

​Despite their remarkable achievements, modern Large Language Models (LLMs) encounter exorbitant computational and memory footprints. Recently, several works have shown significant success in training-free and data-free compression (pruning and quantization) of LLMs achieving 50-60% sparsity and reducing the bit-width down to 3 or 4 bits… Read More »Compressing LLMs: The Truth is Rarely Pure and Never Simple Apple Machine Learning Research

​[[{“value”:”Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation… Read More »Poly-View Contrastive Learning Apple Machine Learning Research

​We investigate the capabilities of transformer models on relational reasoning tasks. In these tasks, models are trained on a set of strings encoding abstract relations, and are then tested out-of-distribution on data that contains symbols that did not appear in the training dataset. We prove… Read More »When can transformers reason with abstract symbols? Apple Machine Learning Research