You can check the request to detect whether you're in a Drupal modal:
$request->query->get('_wrapper_format') === 'drupal_modal';
I needed to use this when embedding a block plugin within a modal form, and wanted to display the block slightly differently than it normally gets displayed.