Set POETRY_VIRTUALENVS_CREATE to false.
POETRY_VIRTUALENVS_CREATE
false
This will disable the creation of a virtual env during poetry install even if HOME changes (Github Actions does this).
poetry install
HOME