From f0e2f5319a4076c0cdaeca464ad30f015c08d065 Mon Sep 17 00:00:00 2001 From: Stephen Gildea Date: Tue, 25 Apr 2023 22:29:05 -0700 Subject: [PATCH 1/7] New methods to access mailbox.Maildir message info and flags: get_info, set_info, get_flags, set_flags, add_flag, remove_flag. These methods speed up accessing a message's info and/or flags and are useful when it is not necessary to access the message's contents, as when iterating over a Maildir to find messages with specific flags. --- Doc/library/mailbox.rst | 104 ++++++++++++++++++++++++++++++++++++++- Lib/mailbox.py | 42 ++++++++++++++++ Lib/test/test_mailbox.py | 86 ++++++++++++++++++++++++++++++++ Misc/ACKS | 1 + 4 files changed, 232 insertions(+), 1 deletion(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index 56908dedea1b40..00489a63018ed4 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -424,6 +424,108 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. remove the underlying message while the returned file remains open. + .. method:: get_flags(key) + + Return as a string the flags that are set on the message + corresponding to *key*. + This is the same as ``get_message(key).get_flags()`` but much + faster, because it does not open the message file. + Use this method when iterating over the keys to determine which + messages are interesting to get. + + If you do have a :class:`MaildirMessage` object, use + its :meth:`~MaildirMessage.get_flags` method instead, because + changes made by the message's :meth:`~MaildirMessage.set_flags`, + :meth:`~MaildirMessage.add_flag` and :meth:`~MaildirMessage.remove_flag` + methods are not reflected here until the mailbox's + :meth:`__setitem__` method is called. + + .. versionadded:: 3.12 + + + .. method:: set_flags(key, flags) + + On the message corresponding to *key*, set the flags specified + by *flags* and unset all others. + Calling ``some_mailbox.set_flags(key, flags)`` is similar to :: + + one_message = some_mailbox.get_message(key) + one_message.set_flags(flags) + some_mailbox[key] = one_message + + but faster, because it does not open the message file. + + If you do have a :class:`MaildirMessage` object, use + its :meth:`~MaildirMessage.set_flags` method instead, because + changes made with this mailbox method will not be visible to the + message object's method, :meth:`~MaildirMessage.get_flags`. + + .. versionadded:: 3.12 + + + .. method:: add_flag(key, flag) + + On the message corresponding to *key*, set the flags specified + by *flag* without changing other flags. To add more than one + flag at a time, *flag* may be a string of more than one character. + + Considerations for using this method versus the message object's + :meth:`~MaildirMessage.add_flag` method are similar to + those for :meth:`set_flags`; see the discussion there. + + .. versionadded:: 3.12 + + + .. method:: remove_flag(key, flag) + + On the message corresponding to *key*, unset the flags specified + by *flag* without changing other flags. To remove more than one + flag at a time, *flag* may be a string of more than one character. + + Considerations for using this method versus the message object's + :meth:`~MaildirMessage.remove_flag` method are similar to + those for :meth:`set_flags`; see the discussion there. + + .. versionadded:: 3.12 + + + .. method:: get_info(key) + + Return a string containing the info for the message + corresponding to *key*. + This is the same as ``get_message(key).get_info()`` but much + faster, because it does not open the message file. + Use this method when iterating over the keys to determine which + messages are interesting to get. + + If you do have a :class:`MaildirMessage` object, use + its :meth:`~MaildirMessage.get_info` method instead, because + changes made by the message's :meth:`~MaildirMessage.set_info` method + are not reflected here until the mailbox's :meth:`__setitem__` method + is called. + + .. versionadded:: 3.12 + + + .. method:: set_info(key, info) + + Set the info of the message corresponding to *key* to *info*. + Calling ``some_mailbox.set_info(key, flags)`` is similar to :: + + one_message = some_mailbox.get_message(key) + one_message.set_info(info) + some_mailbox[key] = one_message + + but faster, because it does not open the message file. + + If you do have a :class:`MaildirMessage` object, use + its :meth:`~MaildirMessage.set_info` method instead, because + changes made with this mailbox method will not be visible to the + message object's method, :meth:`~MaildirMessage.get_info`. + + .. versionadded:: 3.12 + + .. seealso:: `maildir man page from Courier `_ @@ -838,7 +940,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. .. note:: A message is typically moved from :file:`new` to :file:`cur` after its - mailbox has been accessed, whether or not the message is has been + mailbox has been accessed, whether or not the message has been read. A message ``msg`` has been read if ``"S" in msg.get_flags()`` is ``True``. diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 59834a2b3b5243..5bbb8a79216bee 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -395,6 +395,48 @@ def get_file(self, key): f = open(os.path.join(self._path, self._lookup(key)), 'rb') return _ProxyFile(f) + def get_info(self, key): + """Get the keyed message's "info" as a string.""" + subpath = self._lookup(key) + if self.colon in subpath: + return subpath.split(self.colon)[-1] + return '' + + def set_info(self, key, info): + """Set the keyed message's "info" string.""" + if not isinstance(info, str): + raise TypeError('info must be a string: %s' % type(info)) + old_subpath = self._lookup(key) + new_subpath = old_subpath.split(self.colon)[0] + if info: + new_subpath += self.colon + info + if new_subpath == old_subpath: + return + old_path = os.path.join(self._path, old_subpath) + new_path = os.path.join(self._path, new_subpath) + os.rename(old_path, new_path) + self._toc[key] = new_subpath + + def get_flags(self, key): + """Return as a string the flags that are set on the keyed message.""" + info = self.get_info(key) + if info.startswith('2,'): + return info[2:] + return '' + + def set_flags(self, key, flags): + """Set the given flags and unset all others on the keyed message.""" + self.set_info(key, '2,' + ''.join(sorted(flags))) + + def add_flag(self, key, flag): + """Set the given flag(s) without changing others on the keyed message.""" + self.set_flags(key, ''.join(set(self.get_flags(key)) | set(flag))) + + def remove_flag(self, key, flag): + """Unset the given string flag(s) without changing others on the keyed message.""" + if self.get_flags(key): + self.set_flags(key, ''.join(set(self.get_flags(key)) - set(flag))) + def iterkeys(self): """Return an iterator over keys.""" self._refresh() diff --git a/Lib/test/test_mailbox.py b/Lib/test/test_mailbox.py index 4c592eaf34da23..2e22a5ea0d08bf 100644 --- a/Lib/test/test_mailbox.py +++ b/Lib/test/test_mailbox.py @@ -844,6 +844,92 @@ def test_lock_unlock(self): self._box.lock() self._box.unlock() + def test_get_info(self): + # Test getting message info from Maildir, not the message. + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + self.assertEqual(self._box.get_info(key), '') + msg.set_info('OurTestInfo') + self._box[key] = msg + self.assertEqual(self._box.get_info(key), 'OurTestInfo') + + def test_set_info(self): + # Test setting message info from Maildir, not the message. + # This should immediately rename the message file. + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + def check_info(oldinfo, newinfo): + oldfilename = os.path.join(self._box._path, self._box._lookup(key)) + newsubpath = self._box._lookup(key).split(self._box.colon)[0] + if newinfo: + newsubpath += self._box.colon + newinfo + newfilename = os.path.join(self._box._path, newsubpath) + # assert initial conditions + self.assertEqual(self._box.get_info(key), oldinfo) + if not oldinfo: + self.assertNotIn(self._box._lookup(key), self._box.colon) + self.assertTrue(os.path.exists(oldfilename)) + if oldinfo != newinfo: + self.assertFalse(os.path.exists(newfilename)) + # do the rename + self._box.set_info(key, newinfo) + # assert post conditions + if not newinfo: + self.assertNotIn(self._box._lookup(key), self._box.colon) + if oldinfo != newinfo: + self.assertFalse(os.path.exists(oldfilename)) + self.assertTrue(os.path.exists(newfilename)) + self.assertEqual(self._box.get_info(key), newinfo) + # none -> has info + check_info('', 'info1') + # has info -> same info + check_info('info1', 'info1') + # has info -> different info + check_info('info1', 'info2') + # has info -> none + check_info('info2', '') + # none -> none + check_info('', '') + + def test_get_flags(self): + # Test getting message flags from Maildir, not the message. + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + self.assertEqual(self._box.get_flags(key), '') + msg.set_flags('T') + self._box[key] = msg + self.assertEqual(self._box.get_flags(key), 'T') + + def test_set_flags(self): + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + self.assertEqual(self._box.get_flags(key), '') + self._box.set_flags(key, 'S') + self.assertEqual(self._box.get_flags(key), 'S') + + def test_add_flag(self): + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + self.assertEqual(self._box.get_flags(key), '') + self._box.add_flag(key, 'B') + self.assertEqual(self._box.get_flags(key), 'B') + self._box.add_flag(key, 'B') + self.assertEqual(self._box.get_flags(key), 'B') + self._box.add_flag(key, 'AC') + self.assertEqual(self._box.get_flags(key), 'ABC') + + def test_remove_flag(self): + msg = mailbox.MaildirMessage(self._template % 0) + key = self._box.add(msg) + self._box.set_flags(key, 'abc') + self.assertEqual(self._box.get_flags(key), 'abc') + self._box.remove_flag(key, 'b') + self.assertEqual(self._box.get_flags(key), 'ac') + self._box.remove_flag(key, 'b') + self.assertEqual(self._box.get_flags(key), 'ac') + self._box.remove_flag(key, 'ac') + self.assertEqual(self._box.get_flags(key), '') + def test_folder (self): # Test for bug #1569790: verify that folders returned by .get_folder() # use the same factory function. diff --git a/Misc/ACKS b/Misc/ACKS index 19475698a4bc37..7936c04c9818cb 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -624,6 +624,7 @@ Dinu Gherman Subhendu Ghosh Jonathan Giddy Johannes Gijsbers +Stephen Gildea Michael Gilfix Julian Gindi Yannick Gingras From 77ea07bba6a89721fafa65a5f99538d18cc0988d Mon Sep 17 00:00:00 2001 From: Stephen Gildea Date: Wed, 26 Apr 2023 16:37:07 -0700 Subject: [PATCH 2/7] NEWS entry for gh-90890, new mailbox.Maildir methods. --- .../Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst diff --git a/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst b/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst new file mode 100644 index 00000000000000..ee2e69eb27980f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst @@ -0,0 +1,7 @@ +New methods :meth:`mailbox.Maildir.get_info`, +:meth:`mailbox.Maildir.set_info`, :meth:`mailbox.Maildir.get_flags`, +:meth:`mailbox.Maildir.set_flags`, :meth:`mailbox.Maildir.add_flag`, +:meth:`mailbox.Maildir.remove_flag`. These methods speed up accessing a +message's info and/or flags and are useful when it is not necessary to +access the message's contents, as when iterating over a Maildir to find +messages with specific flags. From 9dd04e75630a2cbcb6845680ad02b880db7a5737 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Fri, 10 Nov 2023 16:20:59 -0800 Subject: [PATCH 3/7] versionadded 3.13 --- Doc/library/mailbox.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index 5d89987f02a07b..0d78fec3a01bb1 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -440,7 +440,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. methods are not reflected here until the mailbox's :meth:`__setitem__` method is called. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. method:: set_flags(key, flags) @@ -460,7 +460,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. changes made with this mailbox method will not be visible to the message object's method, :meth:`~MaildirMessage.get_flags`. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. method:: add_flag(key, flag) @@ -473,7 +473,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. :meth:`~MaildirMessage.add_flag` method are similar to those for :meth:`set_flags`; see the discussion there. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. method:: remove_flag(key, flag) @@ -486,7 +486,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. :meth:`~MaildirMessage.remove_flag` method are similar to those for :meth:`set_flags`; see the discussion there. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. method:: get_info(key) @@ -504,7 +504,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. are not reflected here until the mailbox's :meth:`__setitem__` method is called. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. method:: set_info(key, info) @@ -523,7 +523,7 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. changes made with this mailbox method will not be visible to the message object's method, :meth:`~MaildirMessage.get_info`. - .. versionadded:: 3.12 + .. versionadded:: 3.13 .. seealso:: From 01ea3c04cbe1b9baea770a28541ae8fc1dca9742 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Fri, 10 Nov 2023 16:33:10 -0800 Subject: [PATCH 4/7] Add more str type checking --- Lib/mailbox.py | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 5bbb8a79216bee..54fc2e186dca85 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -402,7 +402,7 @@ def get_info(self, key): return subpath.split(self.colon)[-1] return '' - def set_info(self, key, info): + def set_info(self, key, info: str): """Set the keyed message's "info" string.""" if not isinstance(info, str): raise TypeError('info must be a string: %s' % type(info)) @@ -418,22 +418,30 @@ def set_info(self, key, info): self._toc[key] = new_subpath def get_flags(self, key): - """Return as a string the flags that are set on the keyed message.""" + """Return as a string the standard flags that are set on the keyed message.""" info = self.get_info(key) if info.startswith('2,'): return info[2:] return '' - def set_flags(self, key, flags): + def set_flags(self, key, flags: str): """Set the given flags and unset all others on the keyed message.""" - self.set_info(key, '2,' + ''.join(sorted(flags))) + if not isinstance(flags, str): + raise TypeError('flags must be a string: %s' % type(flags)) + # TODO: check if flags are valid standard flag characters? + self.set_info(key, '2,' + ''.join(sorted(set(flags)))) - def add_flag(self, key, flag): + def add_flag(self, key, flag: str): """Set the given flag(s) without changing others on the keyed message.""" + if not isinstance(flag, str): + raise TypeError('flag must be a string: %s' % type(flag)) + # TODO: check that flag is a valid standard flag character? self.set_flags(key, ''.join(set(self.get_flags(key)) | set(flag))) - def remove_flag(self, key, flag): + def remove_flag(self, key, flag: str): """Unset the given string flag(s) without changing others on the keyed message.""" + if not isinstance(flag, str): + raise TypeError('flag must be a string: %s' % type(flag)) if self.get_flags(key): self.set_flags(key, ''.join(set(self.get_flags(key)) - set(flag))) From 04aaa64eaf0966da2040aabf3596b10fa88d7801 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Fri, 10 Nov 2023 16:35:51 -0800 Subject: [PATCH 5/7] modernize to f-strings instead of % --- Lib/mailbox.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 54fc2e186dca85..36afaded705d0a 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -405,7 +405,7 @@ def get_info(self, key): def set_info(self, key, info: str): """Set the keyed message's "info" string.""" if not isinstance(info, str): - raise TypeError('info must be a string: %s' % type(info)) + raise TypeError(f'info must be a string: {type(info)}') old_subpath = self._lookup(key) new_subpath = old_subpath.split(self.colon)[0] if info: @@ -427,21 +427,21 @@ def get_flags(self, key): def set_flags(self, key, flags: str): """Set the given flags and unset all others on the keyed message.""" if not isinstance(flags, str): - raise TypeError('flags must be a string: %s' % type(flags)) + raise TypeError(f'flags must be a string: {type(flags)}') # TODO: check if flags are valid standard flag characters? self.set_info(key, '2,' + ''.join(sorted(set(flags)))) def add_flag(self, key, flag: str): """Set the given flag(s) without changing others on the keyed message.""" if not isinstance(flag, str): - raise TypeError('flag must be a string: %s' % type(flag)) + raise TypeError(f'flag must be a string: {type(flag)}') # TODO: check that flag is a valid standard flag character? self.set_flags(key, ''.join(set(self.get_flags(key)) | set(flag))) def remove_flag(self, key, flag: str): """Unset the given string flag(s) without changing others on the keyed message.""" if not isinstance(flag, str): - raise TypeError('flag must be a string: %s' % type(flag)) + raise TypeError(f'flag must be a string: {type(flag)}') if self.get_flags(key): self.set_flags(key, ''.join(set(self.get_flags(key)) - set(flag))) From 3771862fb873af8d7a12f7c483afa616679450fe Mon Sep 17 00:00:00 2001 From: Stephen Gildea Date: Wed, 13 Dec 2023 13:03:44 -0800 Subject: [PATCH 6/7] Reorder mailbox.Maildir method documentation When new mailbox.Maildir methods were added for 3.13.0a2, their documentation was added at the end of the mailbox.Maildir section instead of grouping them with other methods Maildir adds to Mailbox. This commit moves the new methods' documentation adjacent to documentation for existing Maildir-specific methods, so that the "special remarks" for common methods remains at the end. --- Doc/library/mailbox.rst | 80 ++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/Doc/library/mailbox.rst b/Doc/library/mailbox.rst index 05ffaf6c9b336e..fd60d163378f07 100644 --- a/Doc/library/mailbox.rst +++ b/Doc/library/mailbox.rst @@ -383,46 +383,6 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. last 36 hours. The Maildir specification says that mail-reading programs should do this occasionally. - Some :class:`Mailbox` methods implemented by :class:`Maildir` deserve special - remarks: - - - .. method:: add(message) - __setitem__(key, message) - update(arg) - - .. warning:: - - These methods generate unique file names based upon the current process - ID. When using multiple threads, undetected name clashes may occur and - cause corruption of the mailbox unless threads are coordinated to avoid - using these methods to manipulate the same mailbox simultaneously. - - - .. method:: flush() - - All changes to Maildir mailboxes are immediately applied, so this method - does nothing. - - - .. method:: lock() - unlock() - - Maildir mailboxes do not support (or require) locking, so these methods do - nothing. - - - .. method:: close() - - :class:`Maildir` instances do not keep any open files and the underlying - mailboxes do not support locking, so this method does nothing. - - - .. method:: get_file(key) - - Depending upon the host platform, it may not be possible to modify or - remove the underlying message while the returned file remains open. - .. method:: get_flags(key) @@ -525,6 +485,46 @@ Supported mailbox formats are Maildir, mbox, MH, Babyl, and MMDF. .. versionadded:: 3.13 + Some :class:`Mailbox` methods implemented by :class:`Maildir` deserve special + remarks: + + + .. method:: add(message) + __setitem__(key, message) + update(arg) + + .. warning:: + + These methods generate unique file names based upon the current process + ID. When using multiple threads, undetected name clashes may occur and + cause corruption of the mailbox unless threads are coordinated to avoid + using these methods to manipulate the same mailbox simultaneously. + + + .. method:: flush() + + All changes to Maildir mailboxes are immediately applied, so this method + does nothing. + + + .. method:: lock() + unlock() + + Maildir mailboxes do not support (or require) locking, so these methods do + nothing. + + + .. method:: close() + + :class:`Maildir` instances do not keep any open files and the underlying + mailboxes do not support locking, so this method does nothing. + + + .. method:: get_file(key) + + Depending upon the host platform, it may not be possible to modify or + remove the underlying message while the returned file remains open. + .. seealso:: From 8d39020c06ac1191dc7646eb02e62c03ed7982fd Mon Sep 17 00:00:00 2001 From: Stephen Gildea Date: Wed, 13 Dec 2023 13:11:56 -0800 Subject: [PATCH 7/7] Remove old NEWS.d/next/ blurb --- .../Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst diff --git a/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst b/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst deleted file mode 100644 index ee2e69eb27980f..00000000000000 --- a/Misc/NEWS.d/next/Library/2023-04-26-16-37-00.gh-issue-90890.fIag4w.rst +++ /dev/null @@ -1,7 +0,0 @@ -New methods :meth:`mailbox.Maildir.get_info`, -:meth:`mailbox.Maildir.set_info`, :meth:`mailbox.Maildir.get_flags`, -:meth:`mailbox.Maildir.set_flags`, :meth:`mailbox.Maildir.add_flag`, -:meth:`mailbox.Maildir.remove_flag`. These methods speed up accessing a -message's info and/or flags and are useful when it is not necessary to -access the message's contents, as when iterating over a Maildir to find -messages with specific flags.