Add support for IMAP IDLE in `imaplib` #55454

New issue

Closed

Labels

stdlibtopic-emailtype-feature

ShayRojansky

mannequin

opened

on Feb 18, 2011

· edited by bedevere-app

Mannequin

BPO	11245
Nosy	@warsaw, @ericvsmith, @bitdancer, @vadmium, @mitya57, @soltysh, @ankostis, @jdek
Dependencies	bpo-18921: In imaplib, cached capabilities may be out of date after login
Files	imapidle.patch imapidle.patch: New patch 0001-Add-IDLE-support-to-imaplib.patch: IMAP IDLE patch for imaplib

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-02-18.20:36:10.487>
labels = ['type-feature', 'library', 'expert-email', '3.9']
title = 'Implementation of IMAP IDLE in imaplib?'
updated_at = <Date 2019-12-19.12:58:09.754>
user = 'https://bugs.python.org/ShayRojansky'

bugs.python.org fields:

activity = <Date 2019-12-19.12:58:09.754>
actor = 'jdek'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2011-02-18.20:36:10.487>
creator = 'Shay.Rojansky'
dependencies = ['18921']
files = ['27400', '37555', '48790']
hgrepos = []
issue_num = 11245
keywords = ['patch']
message_count = 24.0
messages = ['128814', '129405', '129407', '171885', '171889', '172365', '172383', '202149', '202342', '202345', '233176', '235972', '236167', '245204', '245246', '245249', '245252', '245253', '245255', '245284', '246235', '246236', '293222', '358678']
nosy_count = 17.0
nosy_names = ['barry', 'pierslauder', 'eric.smith', 'piers', 'r.david.murray', 'Shay.Rojansky', 'martin.panter', 'mitya57', 'maciej.szulik', 'nafur', 'dveeden', 'Malina', 'F.Malina', 'ankostis', 'equaeghe', 'ohreally', 'jdek']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue11245'
versions = ['Python 3.9']

Linked PRs

gh-55454: Add IMAP4 IDLE support to imaplib #122542

ShayRojanskymannequin

MannequinAuthor

IMAP IDLE support is not implemented in the current imaplib. A "drop-in" replacement called imaplib2 exists (), but uses internally managed threads - a heavy solution that is not always appropriate (e.g. when handling many IMAP accounts an asynchronous approach would be more efficient)

I am about to start implementation of an asynchronous select()-compatible approach, and was wondering if there has been any discussion over IDLE, any specific reasons it hasn't been implemented and if eventual integration into imaplib would be a desirable thing.

Proposed approach:

Addition of a new state 'IDLE'
Addition of an idle() method to class IMAP4, which issues the IDLE command to the server and returns immediately. At this point we enter the IDLE state, in which no normal IMAP commands may be issued.
Users can now select() or poll() the socket as they wish
A method can be called to retrieve any untagged responses (e.g. EXISTS) that have arrived since entering the IDLE state. The function returns immediately and does not modify the state.
To end the IDLE state, the user calls a method (done()?) which resumes the previous state.

Would appreciate any sort of feedback...

added

Member

imaplib has no particular maintainer and I know little about it.
Doc says it implements 'a large subset of the IMAP4rev1 client protocol as defined in RFC 2060." I do not remember any discussion on pydev, over the last several years, about imaplib. I presume just the subset was chosen because of some combination of necessity and feasibility, as judged by the implementors. Hence the complement, the unimplemented subset, would be 'not done' rather than 'not wanted'. If your proposed new feature, an IDLE command, is part of this complement, then I would assume that a patch would, in principle, be acceptable.

I cannot comment on your particular proposal, but I hope the above helps as far as it goes.

bitdancer

Member

I just wound up doing a bit of research on this for other reasons. Piers Lauder was the original author of the imaplib module, and he is (as far as I can tell) currently maintaining an imaplib2 module that does support IDLE (but not, I think, python3). But it does IDLE (and other things) via threads, and in the email I found announcing it he didn't think it was suitable for stdlib inclusion (because of the threading). Piers hasn't contributed to core in quite a while as far as I can tell, but he was active in a bug report back in 2008 according to google, so I thought I'd add him to nosy and see if he has time for an opinion.

nafurmannequin

Mannequin

We have implemented this functionality according to RFC 2177.
We actually implemented a synchronous idle function that blocks until a timeout occurs or the server sent some event.

This is not the most flexible way, however it will provide a basic functionality that enables user to use imap idle based notifications.
Besides, every other solution would require threads or regular polling.

See attached patch file.

bitdancer

Member

To fully answer the original question that opened this issue: contributions will be welcomed. While I don't currently have time to work on imaplib myself, I have an interest and will review and if appropriate commit patches.

I like Shay's proposal, but absent a patch along those lines having blocking IMAP support will definitely be an improvement. An application needing to monitor more than one imap connection could do its own threading.

Thanks for proposing the patch. Could you please submit a contributor agreement? I probably won't have time to fully consider the proposed patch for a bit, but I've put it on my todo list.

test_imaplib does have a testing framework now, do you think you could write tests for the new feature?

bitdancer

added

topic-email

on Oct 3, 2012

nafurmannequin

Mannequin

I got the confirmation for my agreement.

I'm not quite sure about the tests, as I'm not really familiar with the way this is done in cpython. The test_imaplib.py seems to cover all ways to connect to some server, but none of the actual imap commands. The patch only implements another commands (whose behaviour is highly/only dependent of other events on the server).

Hence, I don't see a way to create a meaningfull test case other than just calling the command...

bitdancer

Member

Yeah writing a good test case for this is a bit tricky, since we'll need some infrastructure (an Event?) so we can prove that the call has blocked without delaying the test for very long.

I'm not sure when I'll be able to get to this...I'm not going to check it in without a test, so I'll have to find enough time to be able to write the tests.

nafurmannequin

Mannequin

I stumbled about this issue again and would really like to see it fixed.

I see the possibility to create a test case in combination with the first test sequence which creates a temporary mail. Would it be enough, that we just call IDLE in some folder, create a temporary mail in this folder and check if it returns?

Unfortuantely, I have not been able to write code for such a test case yet, as the whole test routine fails with "[PRIVACYREQUIRED] Plaintext authentication disallowed on non-secure (SSL/TLS) connections". This is using 3.2.3, but I guess it will not be any different with the current release... (as it is the same with 2.7.3)

bitdancer

Member

What do you mean by the whole test routine failing? The test suite is currently passing on the buildbots, so are you speaking of the new test you are trying to write?

bitdancer

Member

Hmm. Looking at this again, it appears as though there's no way to interrupt IDLE if you want to, say, send an email. If you are actually using this in code, how are you handling that situation?

nafurmannequin

Mannequin

So, let's resurrect this one.

For the project that lead to the old patch, we did not need this feature.
However, we now needed are more complete implementation of IDLE.
Hence, we extended this to return after sending idle() and support polling, leaving idle mode or wait until something happens (like before).

Malinamannequin

Mannequin

IMAP polling hurts, just merge imaplib2 into standard library as imaplib.

Piers Lauder authored imaplib IMAP4 client, part of python standard library, back in December 1997 based on RFC 2060. In 2003 RFC 2060 was made obsolete by RFC 3501 adding important features and Piers released imaplib2 which receives feature updates since.
Last feature updates to the standard library imaplib were before Piers retired from Sydney University a decade ago.

imaplib2 presents an almost identical API as that provided by the standard library imaplib, the main difference being that imaplib2 allows parallel execution of commands on the IMAP4 server, and implements the IDLE extension, so NO POLLING IS REQUIRED. IMAP server will push new mail notifications to the client. Imaplib2 also supports COMPRESS, ID, better timeout handling etc. There is 975 more lines of code all doing useful things a modern IMAP client needs.

imaplib2 can be substituted for imaplib in existing clients with no changes in the code apart from required logout call to shutdown the threads.

Old imaplib was ported to Python 3 with the rest of the standard library. I am working to port imaplib2 to py3, stuck on receiving bytes v strings.

References:

imaplib2 code and docs
http://sourceforge.net/p/imaplib2/code/ci/master/tree/
also http://sydney.edu.au/engineering/it/~piers/python/imaplib2.html

imaplib
https://hg.python.org/cpython/file/3.4/Lib/imaplib.py

Ruby stdlib support for idle (not that it hurts python performance, just my pride)
http://ruby-doc.org/stdlib-2.0.0/libdoc/net/imap/rdoc/Net/IMAP.html#method-i-idle

33 remaining items

foresto

added a commit that references this issue

on Aug 2, 2024

pythongh-55454: Add IMAP4 IDLE support to imaplib

c7c78e4

foresto

added a commit that references this issue

on Aug 16, 2024

pythongh-55454: Add IMAP4 IDLE support to imaplib

4f25877

foresto

added a commit that references this issue

on Sep 1, 2024

pythongh-55454: Add IMAP4 IDLE support to imaplib

572d1e5

foresto

added 2 commits that reference this issue

on Sep 11, 2024

pythongh-55454: Clarify imaplib idle() docs

9dd2562

pythongh-55454: Clarify imaplib idle() docs

removed

changed the title ~~[-]Implementation of IMAP IDLE in imaplib?[/-]~~ Add support for IMAP IDLE in `imaplib`

on Dec 3, 2024

serhiy-storchaka

Member

I am sorry, but I am not sure the interface proposed in #122542 is the best possible interface. There are some issues with it:

Using the iterator protocol means that the polling is blocking. When we are blocked on waiting a response from the server, there is no way to interrupt waiting. It means that an interactive application which displays an IMAP4 folder status at runtime cannot be responsible to user input. We need a non-blocking operation to poll a response if there is one and return immediately if there are no responses. We need also a blocking operation with timeout and a way to interrupt thread before expiration of the timeout.
Using the context manager protocol means that DONE is only sent after leaving the with block. After client sent DONE, server can still send status updates. What happens to them? You cannot use Idler after leaving the with block. We need a method to send DONE and read status updates that came after this.
Server can send status updates not only for IDLE command, but for other commands. It is common to use the NOOP command for this. There is a large similarity between NOOP and IDLE commands, but is is not seen in the proposed interface. You can read status updates from untagged_responces after the NOOP command, although this is not thread-safe. It would be nice to have unified way to get status updates for IDLE and other commands. It would be nice if it is thread-safe or there is a clear way to use the IMAP4 client in thread-safe way.

Let's look what other solutions provide. Interesting, there were two different third-party packages named imaplib2, both were proposed for inclusion in the stdlib.

Piers Lauder's imaplib2 was discussed above. It is available on PyPI. I has a simple blocking idle() method which waits for first server response or expiration of the timeout. Then it send DONE and wait for tagged response for the IDLE command. You can read status updates from untagged_responces after this. It may be not very efficient, because you need to send IDLE again, but it fits perfectly in the line with other commands. This package is compatible with imaplib. The main disadvantage of this package is that it runs internal threads. But I think that there may be possibility to implement it without running internal thread. Instead the user will need to run external threads to utilize the IMAP4 concurrency features. imaplib will still need to use locks and events to make its methods thread-safe.

Maxim Khitrov's imaplib2 was discussed in Python-ideas. It provides incompatible interface, in particularly concurrent fetch(). Its interface for IDLE is similar to the one proposed in #122542, with support for the context manager and iterator protocols, but it also has the poll() method.

Looking at support in other programming languages, in Perl, support for IDLE is provided by three methods -- idle, idle_data and done. idle is non-blocking, it simply sends the IDLE command, idle_data is non-blocking or blocking with timeout, it polls server response. done sends DONE.

serhiy-storchaka

Member

I think that we first should implement support for concurrent execution of other commands (or at least draw a plan for it). Then we can add support for the IDLE command which fits the general design.

I think that many (if not all) commands should have a non-blocking variant. Either boolean parameter or a method with special suffix. Non-blocking method sends the command but does not wait for the tagged response. It should return something Future-like. You can block a thread waiting on the response for specific command or check if there is response without blocking.

There should be a thread-safe method for polling non-tagged responses (after IDLE or other commands). It should provide blocking and non-blocking interface. Iterator protocol also can be supported for convenience -- you can iterate only already received non-tagged responses or block to wait on new responses.

IDLE will also have blocking and non-blocking variants. In blocking variant it will wait on tagged server response (which can only be sent after client sent DONE in other thread), in non-blocking variant it will return a Future-like object (which also supports the context manager protocol and has the done() method). We can also implement auto-deblocking option like in Piers Lauder's imaplib2 -- send DONE after first non-tagged response and wait for tagged response.

foresto

Contributor

I am sorry, but I am not sure the interface proposed in #122542 is the best possible interface. There are some issues with it:

Using the iterator protocol means that the polling is blocking.

That's not a requirement of the iterator protocol. It supports stopping immediately without producing any results, much like the read() system call can do with a non-blocking file descriptor. As a matter of fact, #122542 already does this when a limited-duration Idler has expired.

It does block in other cases, and this is by design, because the whole of imaplib is a blocking API. #122542 respects and preserves that design. It's a small extension to the existing module, not a replacement for it.

(And just to be clear for the sake of other readers, the IDLE command doesn't poll, at least not in the way we usually mean in computer science. The point of IDLE is to avoid polling, and instead react to events sent by the server as they arrive.)

When we are blocked on waiting a response from the server, there is no way to interrupt waiting.

There are two ways. Waiting can be cleanly interrupted by:

A duration (time limit) passed to the call that sets up the IDLE context.
An exception. For example, a KeyboardInterrupt in an interactive terminal program, or some other exception raised by a timer or signal handler.

If you're wishing for an API to interrupt a blocking IDLE from another thread, then it is true that no such thing exists at the moment. The same is true for every other command in imaplib.

Such a thing could be added in the future. One way would be modifying imaplib to use non-bocking sockets, and make blocking commands call select() on both the IMAP socket and one end of a socket pair. Another thread could then write to the other end of the socket pair to interrupt the command. The interface in #122542 would not have to change.

Of course, moving imaplib to non-blocking sockets and adding calls to interrupt its blocking commands would be additional features, beyond the scope of #122542. This PR simply adds support for IMAP IDLE. It doesn't aim to overhaul or replace imaplib.

It means that an interactive application which displays an IMAP4 folder status at runtime cannot be responsible to user input.

Let's keep in mind that introducing IDLE support would't mean all applications have to use it. They can continue using the existing commands, and poll for updates using NOOP, just as they always have.

When considering applications with asynchronous display updates, let's also remember that all of imaplib is already a blocking API, so the issues around blocking calls are already present. An application can deal with them by running commands in a separate thread. #122542 doesn't change this. The same approach can be used with IDLE.

Note that IMAP doesn't allow other commands during IDLE, so an application wishing to do both at the same time would have to keep two connections open, presumably in separate threads. That's a limitation imposed by the protocol.

(I suppose a library could hide this limitation by automatically switching between command mode and IDLE mode, making the switches transparent to application code, but such a library would be considerably more complex than imaplib.)

We need a non-blocking operation to poll a response if there is one and return immediately if there are no responses.

This doesn't exist in any of imaplib's commands. I agree that it could be useful for certain programming models, but it's a separate concern from adding IDLE support. I hope we can agree that the two should not be conflated.

Are you worried that #122542 might get in the way of future non-blocking support in imaplib? I don't believe it would.

Using the context manager protocol means that DONE is only sent after leaving the with block. After client sent DONE, server can still send status updates. What happens to them? You cannot use Idler after leaving the with block. We need a method to send DONE and read status updates that came after this.

Indeed, the server can send untagged responses after DONE is sent. These are already handled in #122542. They can be retrieved with the response() method, just as they can after other commands.

Server can send status updates not only for IDLE command, but for other commands. It is common to use the NOOP command for this. There is a large similarity between NOOP and IDLE commands, but is is not seen in the proposed interface.

Both can produce untagged responses, but their similarity ends there. Notably:

The IDLE command requests a temporary inversion of control, which is represented nicely in the proposed interface using a python context manager.
The IDLE command produces an unbounded stream of responses with unpredictable timing, which is represented nicely in the proposed interface using python iteration.

IDLE and NOOP are made for different situations, and their interfaces reflect that.

You can read status updates from untagged_responces after the NOOP command,

The untagged_responses attribute is an internal, undocumented interface. Programs should not be using it. The correct way to do this is through the response() method. This still works in #122542, including responses after leaving the IDLE context (as mentioned above).

It would be nice to have unified way to get status updates for IDLE and other commands.

This could be done with a callback interface, iteration, or polling. (Although using a polling interface for IDLE would be a bit silly, because its purpose is to avoid polling; that's what NOOP is for.)

However, it would mean modifying or duplicating every command method in imaplib, not just IDLE, and adding features that are orthogonal to the present issue. I think that would make more sense as a separate feature request. Don't you?

It would be nice if it is thread-safe or there is a clear way to use the IMAP4 client in thread-safe way.

I guess you mean sharing a single IMAP4 object and connection among multiple threads. (As far as I know, imaplib is already thread-safe when each thread uses its own IMAP4 object. I don't see any writes to global variables, at least.)

I think this would be a little tricky even without IDLE, since you would have to multiplex commands from different threads into a single connection, ensure atomic writes to the server, sort out the various tagged and untagged responses, and route them to the threads that wanted them. It would be even harder with IDLE, because IMAP doesn't support command mode and IDLE mode at the same time. You would have to dynamically switch between modes within the library while simulating a single mode from each thread's point of view.

I suppose that might be possible, but would be a major undertaking, and the result would arguably not be imaplib any more. If it's important to you, I think you should write up a separate issue for it. This one is about adding IDLE support to imaplib.

Let's look what other solutions provide. Interesting, there were two different third-party packages named imaplib2, both were proposed for inclusion in the stdlib.

Piers Lauder's imaplib2 was discussed above. It is available on PyPI. I has a simple blocking idle() method which waits for first server response or expiration of the timeout. Then it send DONE and wait for tagged response for the IDLE command. You can read status updates from untagged_responces after this. It may be not very efficient, because you need to send IDLE again, but it fits perfectly in the line with other commands. This package is compatible with imaplib. The main disadvantage of this package is that it runs internal threads. But I think that there may be possibility to implement it without running internal thread. Instead the user will need to run external threads to utilize the IMAP4 concurrency features. imaplib will still need to use locks and events to make its methods thread-safe.

I've seen that one. #122542 already does all those things, and more, and it does them more elegantly, and without imposing threads on applications.

Maxim Khitrov's imaplib2 was discussed in Python-ideas. It provides incompatible interface, in particularly concurrent fetch(). Its interface for IDLE is similar to the one proposed in #122542, with support for the context manager and iterator protocols, but it also has the poll() method.

I've seen that one, and no, its interface for IDLE is not similar to #122542. Note that its iteration feature doesn't work during IDLE. Instead, it forces the application to use a polling loop and make select() or sleep() calls in between. It also has various implementation flaws that will make it misbehave in conditions where #122542 works correctly. Perhaps those could be fixed, but there's a bigger problem with it:

The interface is incompatible with imaplib, as you pointed out. So adopting it would effectively mean replacing imaplib.

Looking at support in other programming languages, in Perl, support for IDLE is provided by three methods -- idle, idle_data and done. idle is non-blocking, it simply sends the IDLE command, idle_data is non-blocking or blocking with timeout, it polls server response. done sends DONE.

My takeaway from everything you've written here is that you would like a non-blocking IMAP client library with polling semantics.

I can see ways in which that would be useful, but imaplib is not that library, and there is no IDLE design or implementation that can change that. The imaplib API is a blocking design throughout. Getting what you want would require substantial changes all across the library, or else replacing it.

You're not alone in your wish. An imaplib overhaul or replacement has been suggested every so often for at least 13 years. However, nobody has done the work to make it happen in the standard library, and there is no sign that anyone will have done that work any time soon.

Meanwhile, we finally have a clean, working, compatible, pythonic, documented, reviewed pull request providing IDLE support, which people have also been requesting for at least 13 years. And as far as I can tell, there is nothing about it that would prevent the non-blocking interface you want from being added later.

I therefore propose that the request for non-blocking imaplib interfaces (or replacing imaplib entirely) be moved to a new issue.