calling a common function that uses a db session from inside fastapi as well as celery

[sandys]

hi guys,
I have a pretty common usecase that im not able to figure out how to do - i have a common function that i need to call using fastapi as well as a celery worker.

However, im not able to figure out how to create the session in a way that it will work for both api and worker.

the code that is in this repo does not have an example which i could find, but it is super common in real world.

Could you guys comment on how we should write this ? If you could just add a function that calls a db query and use that function in both the api and celery worker, that should be enough.

we have a pretty hacky way of doing this in flask - check if a flask request context exists, etc etc . I'm hoping there's a much more cleaner way here.

[ghost]

Hi sandys,

You can import Session from app.db.session in your worker.py file and use it inside your tasks.

Cheers

[sandys]

That will import fastapi libraries inside celery code. That is the part that I want to avoid

…

On Thu, 3 Oct, 2019, 14:10 driribarne, ***@***.***> wrote: Hi sandys, You can import Session from app.db.session in your worker.py file and use it inside your tasks. Cheers — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#68>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU2AAI6MWTEYLM4EQWLQMWVWLANCNFSM4IZBZYIA> .

[dmontagu]

Just put the session creation function in a self contained module that doesn't import fastapi code.

Then import that function into your fastapi code, and into your celery code.

[sandys]

hi, so we have attempted to do that - the big question for us is where do we connect a fastapi session to a sqlalchemy session ultimately we tried to create db session in the middleware - https://github.com/tiangolo/full-stack-fastapi-postgresql/blob/b96d78db99d8c4f37f0df64df587ffebda4bfc56/%7B%7Bcookiecutter.project_slug%7D%7D/backend/app/app/main.py#L31-L36 we are hoping this means that fastapi will create a new database session for every fastapi-session/request . I hope this is correct ? the session itself is created like this - https://github.com/tiangolo/full-stack-fastapi-postgresql/blob/170231783adddb3ba974bfb98dc66116b95ee6af/%7B%7Bcookiecutter.project_slug%7D%7D/backend/app/app/db/session.py we replicate this in our workers (we havent found a clean way to import this code without bringing in fastapi dependencies) would appreciate any feedback.

…

On Thu, Oct 3, 2019 at 3:15 PM dmontagu ***@***.***> wrote: Just put the session creation function in a self contained module that doesn't import fastapi code. Then import that function into your fastapi code, and into your celery code. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#68>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU6HDRC5EHZGG3NC4WLQMW5K5ANCNFSM4IZBZYIA> .

[dmontagu]

we are hoping this means that fastapi will create a new database session
for every fastapi-session/request .
I hope this is correct ?

Yes that's right.

I see two options.

---

Option 1: Move the config to a self-contained file that doesn't have fastapi as a dependency. Then the session.py module could be imported safely. This might not be desirable if the only setting you need in the workers is the sqlalchemy uri though.

---

Option 2: Make a get_sessionmaker_instance function that takes a URI as an argument:

from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker

# from app.core import config  # no longer necessary

def get_sessionmaker_instance(uri: str) -> sessionmaker:
    engine = create_engine(uri, pool_pre_ping=True)
    db_session = scoped_session(
        sessionmaker(autocommit=False, autoflush=False, bind=engine)
    )
    Session = sessionmaker(autocommit=False, autoflush=False, bind=engine)
    return Session

Then, instead of importing Session from this module, you import the function, and optionally create a Session in an app-specific file via
Session = get_sessionmaker_instance(config.SQLALCHEMY_DATABASE_URI), and do the same in worker code. Note -- to avoid reducing to option 1, you'll need a way to get the SQLALCHEMY_DATABASE_URI that doesn't depend on the config file. But that's going to be unavoidable if you are explicitly trying to not share dependencies that you want to use while generating your config.

[sandys]

thanks for replying - im basically doing the second ...but your code was nicer. Well I'm going to be importing the same config file from both fastapi and celery (this gives context to another bug about loading config via starlette vs pydantic that i filed as well). This is unavoidable - we integrate with about a hundred external services and they all exist in staging + production configuration. thanks! rega

…

On Thu, Oct 3, 2019 at 3:50 PM dmontagu ***@***.***> wrote: we are hoping this means that fastapi will create a new database session for every fastapi-session/request . I hope this is correct ? Yes that's right. I see two options. ------------------------------ Option 1: Move the config to a self-contained file that doesn't have fastapi as a dependency. Then the session.py module could be imported safely. This might not be desirable if the only setting you need in the workers is the sqlalchemy uri though. ------------------------------ Option 2: Make a get_sessionmaker_instance function that takes a URI as an argument: from sqlalchemy import create_enginefrom sqlalchemy.orm import scoped_session, sessionmaker # from app.core import config def get_sessionmaker_instance(uri: str) -> sessionmaker: engine = create_engine(uri, pool_pre_ping=True) db_session = scoped_session( sessionmaker(autocommit=False, autoflush=False, bind=engine) ) Session = sessionmaker(autocommit=False, autoflush=False, bind=engine) return Session Then, instead of importing Session from this module, you import the function, and optionally create a Session in an app-specific file via Session = get_sessionmaker_instance(config.SQLALCHEMY_DATABASE_URI), and do the same in worker code. Note -- to avoid reducing to option 1, you'll need a way to get the SQLALCHEMY_DATABASE_URI that doesn't depend on the config file. But that's going to be unavoidable if you are explicitly trying to not share dependencies that you *want* to use while generating your config. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#68>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYUYP4OBFSQUIDIFKY4DQMXBOPANCNFSM4IZBZYIA> .

[dmontagu]

@sandys What is the starlette vs pydantic config bug? I'm curious

[sandys]

fastapi/fastapi#508 (comment) you had given a good hint there. im wondering if you would change your answer given this one.

…

On Thu, Oct 3, 2019 at 4:06 PM dmontagu ***@***.***> wrote: @sandys <https://github.com/sandys> What is the starlette vs pydantic config bug? I'm curious — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU4BXLNN5BTQC52NWSDQMXDKHANCNFSM4IZBZYIA> .

[tiangolo]

Thanks for all the help here @dmontagu ! 🙇 🍰

@sandys I see you posted a link to another post asking about configs, here are the new FastAPI docs for configs and settings: https://fastapi.tiangolo.com/advanced/settings/

Thanks for reporting back and closing the issue 👍

[sandys]

closed with thanks