If you head to your Marketing>Reports, there is a report called the Email Status Search. I would recommend opening this report, and running a search for last open date range of a date prior to your start with Infusionsoft - today. You can also add the ‘Last Open Date’ column via the columns tab in your search criteria window. Running this search will present a list of contacts with a Last Open Date on record. From here, select all contacts, and give them a temporary tag (Ex: OpenedEmail) via an ‘Action’
Then, head over to your CRM>Contacts to run a simple search of contacts who ‘DO NOT CONTAIN’ the temporary tag you applied to the contacts with a last open date. This will pull your list of contacts that do not have a last open date on record. From here, I would recommend possibly selecting all contacts, and exporting all their available data, before reviewing and deciding to delete them.