A camera acquires an image of an item involved in an interaction, such as the item being picked or placed on a shelf by a user. The item depicted in the image is identified via item recognition using a customized neural network that uses previously trained subnetworks for a set of candidate items. The candidate items are determined based on items near a user location, items associated with a user cart, items in a shopping list or wish list associated with the user, or items otherwise associated with the user. Each previously trained subnetwork for a candidate item includes one or more reference images and corresponding reference feature data. The customized neural network uses a matching and inference network to test the image against the candidate items and provide comparison data indicative of a similarity between the item depicted in the image and one of the candidate items.