Enqueue with MessageId

[dpvreony]

Hi

We have a scenario where we need to support duplicate detection for messages being enqueued which is done by messageid https://msdn.microsoft.com/library/azure/microsoft.servicebus.messaging.brokeredmessage.messageid.aspx

I've done the basics in a fork (dpvreony-forks@50f18ce) but it is currently off the back of #71

Thanks

[niemyjski]

What are your plans for supporting dedup on the message bus? Would it be something built in to foundatio or just relying on existing service bus implementations (that may have it) I think this would be really nice to have.

I think we may want to do some more research on our model in general and add this field and maybe others. Would you mind taking a look at some of the popular message buses and see if there is anything else we may be missing that would be useful as well.

[dpvreony]

Apologies I've been a bit busy. I have done some pieces around how we're using duplicate detection and options for implementation in something like foundatio. I will try and get something written up ASAP.

Regards

[niemyjski]

Thanks for the head up, we appreciate it.

[niemyjski]

@dpvreony @ejsmith is this something we still need? Will it open up any other scenarios like maybe dedup?

[brutaldev]

Duplicate detection in the Azure Service bus is very important to me as well. We've built a very similar set of core libraries for queuing, logging, caching etc and want to switch over to Foundatio specifically for the non-Azure support. Using MessageId is a critical blocker right now as well as bulk messaging which I'll open another case for. There is currently no ability to optimize the settings for the service bus topics either (another case as well) so you can't even enable duplicate detection or the amount of time to detect.

You would need a custom de-dupe implementation for RabbitMQ since there is nothing built in (that I know of). There are hacky solutions for Redis messaging by using properties that would ensure uniqueness (correlation) the amount of time to keep the duplicate detection can simply be the TTL if it's not dequeue. Long story short, it's not going to be easy to introduce this and provide parity with the other concrete implementations :(

[niemyjski]

@brutaldev we'd love a pull request for this (please use the cloudwatch branch until we merge it in). I've added a bunch of fixes and settings for those options in the cloud watch branch. We just need a reliable way to create unique message ids. If you have ideas on this, let us know. We can take a stab at this in one of them and support the others like rabbitmq later.

[brutaldev]

@niemyjski The developer would need to set the message ID manually to something they deem to be "unique" in their system and it gets ignored for the configured amount of time. I would see it as an error if the message ID is not set when the queue/topic has duplicate detection enabled. This leaves it up to the developer to generate a random ID if they want to ignore the duplicate detection they have enabled.

I would not have duplicate detection enabled at all times, but I would allow the developer to set the message ID at all times since this is not only used for duplicate detection, but often a way to tie the queue item back to your system.

I'll be setting up custom local builds to Foundatio when I get around to replacing our own implementation and make sure I get all the necessary changes in and PR them at the same time. I'll need to battle test my branch in production first.

[ejsmith]

This would be super easy to implement using our throttling lock provider.

[niemyjski]

I think we could keep it pretty simple too, you could just have an interface marker that has message id on it and we could just look at the model you are passing in and get the message id from that, you could go a step further and just mark that field to not be serialized with an attribute so you don't take the payload hit :)

[ejsmith]

Queue entries already support metadata. I wonder if we should expose the ability to enqueue those as well as the queue models themselves. Because ideally this would just be reading the information from that metadata.