Support Retain for Publish

[PaulKierstead]

One useful pattern to track device up/down status is to have a will set to .../status with value '0' and retain, and when the device comes up to publish to the .../status with value '1' with retain. The retain allows the status to be interrogated at anytime by a client. The library has retain flag for will, but not for publish. Of course, it would be useful for lots of other things too, but thought I would point out one very valuable one...

[MichMich]

+1 for retain support

[toddtreece]

if you are using this library with adafruit IO, your last published value should be retained by default on the server. we will try to get this added to the library as well.

[probonopd]

Possibly related to https://forums.adafruit.com/viewtopic.php?f=56&t=88407 where I asked for functionality so that "the ESP8266 would not have to wait until a feed is updated; instead it would ask adafruit.io "give me the last value of feed x".

The answer given by @jwcooper was:

I believe this should be the default scenario. You should get an initial value as you subscribe to a feed. We store the last values sent (once you have sent an initial value, of course) so that you should immediately receive that value.

However, it is unclear how to do this and this ticket, especially the answer by @toddtreece makes me wonder whether it is implemented at all.

[ladyada]

a lot has changed in the last few weeks - ill ask todd again, if he says it can be done, ill add it :)

[PaulKierstead]

@toddtreece answer makes me think there maybe misunderstanding of the usage of the retain bit. The retain bit is not telling the server to keep or not keep the data (it is free to keep as much as it likes), but is telling the server how to handle new subscribers. So, when you subscribe to a topic, if the publisher had set retain, the subscriber immediately gets the last published value (regardless of age). If the publisher did not set retain, a subscriber will only get a message when the publisher publishes a new value. This is a very important distinction, as it lets the publisher determine if the data is good when it is stale. For something like "device.up" (with a retained will value of say "0", and publish of "1" on boot) retain is crucial. For something motion detection, it would be problematic, you likely only want to know a bout motion as it happens, not if it happened some time in the past. Unfortunately MQTT won't give you the age of a message, so sometimes you'll want a retained value (say temperature), but also want to use a "up" topic to make sure the device is still publishing and it isn't days old or something.

In any case, retain is pretty crucial for sensor use in a general purpose broker.

[toddtreece]

@PaulKierstead there is no misunderstanding. we chose to force the retain flag on by default so that people have an easier time getting access to the last value. you bring up a valid point about the stale values, and there are currently ways to check staleness via MQTT.

Adafruit IO processes each message, saves a new record in Postgres, and then sends out new copies of the saved message on multiple MQTT topics with the retain bit flipped (including the full record in JSON format with timestamp at {username}/feeds/{id}/json).

Adafruit IO is independent of this library, and the discussion about IO's behavior with regard to retain can be continued in the io-issues repo. with my earlier comment I was saying is that retain support would have to be added to this library because IO's forced retained behavior is not standard behavior for general purpose MQTT brokers.

[igg2]

Even if Adafruit IO sets the retain flag by default, it doesn't make sense for the retain flag to be a per-connection setting lumped together with will() in this library. Typically this is implemented as a per-feed setting in the publish() call. Arguably, it could also be a subscribe() option, which makes sense given the default IO behavior. As implemented now, even though retain is on by default in IO, clients that connect through this library don't get the last value unless they call will(), which is not intuitively the same setting.

[toddtreece]

@igg2 retain is not set per connection in this library. i think what you might be referring to is the retain flag for the MQTT will feature, which is part of the connect packet and is unrelated to this issue: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/csprd02/mqtt-v3.1.1-csprd02.html#_Toc385349234

the retain feature for publish is not implemented in this library yet, and MQTT does not support setting a retain flag for subscriptions. there might be a separate bug that is causing the forced retained messages from IO to be missed by this library when connecting. we will check into that.

[igg2]

I see, so the retain message in will() refers only to the will message. That makes more sense. Since Adafruit IO has retain on by default for all messages, we should get the last known value for every feed we subscribe to when we connect - is that correct? That's definitely not happening with the latest release from GitHub. I will switch to the master branch and try with that. I know I can work around by getting the last known value from the REST API, but I'd rather keep it all in MQTT if possible.

[ladyada]

ok just pushed to the library, if you subscribe to a topic, adafruit.io will send you the retained message the first time you call readSubscription
however, right now it only works for a single sub. for more subs what we really have to do is use callbacks so the packet isnt 'eaten'
anyways, it is commited - plz try!

[igg2]

So that totally works, except that now if you do have more than one subscription, it fails in mqtt.connect() with "Failed to subscribe". That seems kind of harsh, since now you're effectively limited to one subscription. Having one of them get a retained update at random when first connecting seems better. Better still would be a callback. But wouldn't it be simpler to just have a readRetained() method to return an array of subscription pointers, and leave everything else as it was for now? It only makes sense to get retained values once between connect() and readSubscription() anyway.

[probonopd]

@igg2 do you have an example sketch where Retain can be seen in action? Do I have to modify the esp8266 example sketch for it to use Retain?

[igg2]

You don't have to modify the example sketch (mqtt_esp8266) for it to use retain, it'll just do it, assuming you're using the current master branch (with commit 79dd624). I don't think there's a way to not use retain at this point. Just keep hitting the reset button, and you'll see that it always gets the last value when it first calls readSubscription().

[probonopd]

Thanks @ladyada works for me.

[ladyada]

the real trick is to have it work for multiple subs. i can do it but requires a refactor to also do callback support (e.g. turn readpacket into readandprocesspacket). its on the list to do!