When sensitive to olives, the most conservative approach is to avoid all forms of olive, including olive oil.
When sensitive to olives, can I eat olive oil?
Adair