Feature request: receive and send audio message

[MachengShen]

Hi, Thanks for the nice OA puppet. I'm interested in receiving and sending audio messages with wechat users through OA. Currently the puppet.messageSendFile() method throws an 'unsupported file exception' when trying to send SILK encoded audio files, or .mp3, and the 'EventMessagePayload' also does not receive audio input from the user.
Would appreciate it if we can add this new feature, please instruct how to do it. Thanks!

[wj-Mcat]

Yes, video messagge is not supported now, and pr is welcome. I believe this feature is simple for you. You can have a try so you can become a wechaty contributor 👍
Please refer to :

• message material in official-account
• sendFile method
• messageSend method

[huan]

We now support very limit message types:

https://github.com/wechaty/wechaty-puppet-official-account/blob/381ffb820fcc63e4b89a99c433b696e790e06b7a/src/official-account/webhook.ts#L241-L244

In order to support receiving more message types, like audio, you need to look at:

https://github.com/wechaty/wechaty-puppet-official-account/blob/381ffb820fcc63e4b89a99c433b696e790e06b7a/src/official-account/webhook.ts#L247-L252

I hope the above information can help you and please feel free to let me know if you have more questions. @MachengShen

[MachengShen]

Thanks! I added 'voice' to the 'knownTypeList', and the voice message triggers the onMessage() method in ding-dong.ts, but the message type is 'text' instead of 'audio'. The log shows that the received message was 'voice' before 'const msgPayload = await puppet.messagePayload(payload.messageId)' was called, so my understanding is that something in the wechaty-puppet repo is messing up with the payload?

I attached the code piece to reproduce this issue and the log (level 'silly') from the console:

Steps for reproduce:

ding-dong.ts:

  async function onMessage (payload: EventMessagePayload) {
    const msgPayload = await puppet.messagePayload(payload.messageId)
    log.verbose('ding-dong', `onMessage: ${JSON.stringify(msgPayload)}`)
  }

webhook.ts:

   const knownTypeList = [ 
     'text', 
     'image', 
     'voice',
   ] 

Then send a voice message to the official account

Logs:

12:33:26 VERB Webhook appPost({url: /?signature=4729e1e02e26de6bf16eefa036a7a66ce1fca448&timestamp=1612326804&nonce=1097261724&openid=ozm_X6eCQ_4Z-r1xcEKmLLR_P5cA} with payload: {"ToUserName":"gh_89b011c540f8","FromUserName":"ozm_X6eCQ_4Z-r1xcEKmLLR_P5cA","CreateTime":"1612326804","MsgType":"voice","MediaId":"_0iIuUnF6_UbjOGWYQeScoxKskhaeFdtIz2C2WcT86wTp-ij7vdukdrUyhu-CZ9i","Format":"amr","MsgId":"23082968392782987","Recognition":""}
12:33:26 VERB PayloadStore setMessagePayload(23082968392782987, {"ToUserName":"gh_89b011c540f8","FromUserName":"ozm_X6eCQ_4Z-r1xcEKmLLR_P5cA","CreateTime":"1612326804","MsgType":"voice","MediaId":"_0iIuUnF6_UbjOGWYQeScoxKskhaeFdtIz2C2WcT86wTp-ij7vdukdrUyhu-CZ9i","Format":"amr","MsgId":"23082968392782987","Recognition":""})
12:33:26 VERB Puppet messagePayload(23082968392782987)
12:33:26 SILL Puppet messagePayloadCache(23082968392782987) cache MISS
12:33:26 VERB PuppetOA messageRawPayload(23082968392782987)
12:33:26 VERB PayloadStore getMessagePayload(23082968392782987)
12:33:26 SILL Puppet messagePayload(23082968392782987) cache SET
12:33:26 VERB ding-dong onMessage: {"fromId":"ozm_X6eCQ_4Z-r1xcEKmLLR_P5cA","id":"23082968392782987","timestamp":1612326804,"toId":"gh_89b011c540f8","type":7}

[huan]

Congratulations! It seems that you can receive the right webhook payload from WeChat Official Server when you sent an audio message to your Official Account!

However, it seems that the Wechaty Message Payload has not been set correctly. In order to make it correct, you need to understand the message processing flow in Wechaty Puppet.

Wechaty Puppet Message Processing Flow

1. The Webhook get called by the Tencent Server (what you have done already)

2. The message event will be propagated from the Webhook class to the OfficialAccount class:

   https://github.com/wechaty/wechaty-puppet-official-account/blob/381ffb820fcc63e4b89a99c433b696e790e06b7a/src/official-account/official-account.ts#L116-L119

3. The message event will be propagated from the OfficialAccount class to the PuppetOA class:

   https://github.com/wechaty/wechaty-puppet-official-account/blob/381ffb820fcc63e4b89a99c433b696e790e06b7a/src/puppet-oa.ts#L188-L189

4. After message event be propagated from the PuppetOA to Wechaty, then the puppet.messagePayload() will be called to get the Wechaty Message Payload. Here is the most important part: we need to construct a Wechaty Message Payload from the Raw Message Payload (source code at here):

   const rawPayload = await this.messageRawPayload(messageId)
   const payload    = await this.messageRawPayloadParser(rawPayload)

What to Do Next

So it will be very clear that, what we need to do to support the new Message Type (audio in this case), is to implement the messageRawPayload methods, which you can find here:

https://github.com/wechaty/wechaty-puppet-official-account/blob/381ffb820fcc63e4b89a99c433b696e790e06b7a/src/puppet-oa.ts#L456-L478

I hope the above explanation could help you to move forward, please feel free to let me know if you have more questions.

[huan]

Review for #21

Receiving Audio File

The Wechaty will call puppet.messageFile(id) to get the audio file from a message, so you need to implemented the messageFile() method in puppet. See:

• Puppet Abstract: https://github.com/wechaty/wechaty-puppet/blob/763e94194fd1104007fccad4ba4994365890cde8/src/puppet.ts#L803
• Wechaty Message: https://github.com/wechaty/wechaty/blob/8f8ae6a6cedb3bb746201d9905fbfa995fa325cc/src/user/message.ts#L952

Sending Audio File

The Wechaty will call message.say(fileBox) to send audio message, which will call puppet.messageSendFile(fileBox) under the hood. So you need to implemented the messageSendFile() method in puppetOA. See:

• Puppet Abstract: https://github.com/wechaty/wechaty-puppet/blob/763e94194fd1104007fccad4ba4994365890cde8/src/puppet.ts#L809
• Wechaty Message: https://github.com/wechaty/wechaty/blob/8f8ae6a6cedb3bb746201d9905fbfa995fa325cc/src/user/message.ts#L560-L565

[zhangfand]

Thanks for your prompt reply.

One issue I ran into when trying to update messageSendFile() is mime type. The function uses mime type to determine OAMediaType, and bails out if the mime type is not supported.

https://github.com/wechaty/wechaty-puppet-official-account/blob/308bcb53651aa88d3f14eda2baa50a97afc11632/src/puppet-oa.ts#L531-L545

FileBox.mimeType is inferred from the name of the file. However, the depending mime package fails to infer the expected type audio/AMR from extension name .amr

https://github.com/huan/file-box/blob/master/src/file-box.ts#L270

What would you suggest to work around this?

[huan]

In my understanding, our problem is that the mime package can not get the right type for the .amr file?

My suggestion would like:

1. File an issue to the mime package to track this problem, and
2. Create an issue and workaround PR in file-box module, to set the right mime type for .amr file, and link the issue to the mime package issue for tracking.
3. Create a PR for the mime package for fixing this problem from the root.

Please let me what you think about this solution, thanks.

[huan]

Update

It seems that the mime package has a mime.define() method to add a new mime type to the module.

So we can add the following line to FileBox which I believe could fix the mime issue for amr.

// Require Mime class
const Mime = require('mime/Mime');

// Define mime type -> extensions map
const typeMap = {
  'audio/amr': ['amr'],
};

// Create and use Mime instance
const myMime = new Mime(typeMap);
myMime.getType('amr');            // ⇨ 'audio/amr'

See: https://github.com/broofa/mime#mime-api

[zhangfand]

@huan Thanks for adding support for AMR to FileBox. The solution makes sense.

I have opened an PR in mime-db (the data source backing mime) to add audio/amr support. I will update the thread once that is landed.

[huan]

I saw it has been merged already, cheers!

Please link the issues together in the future, for example, ref to this issue to jshttp/mime-db#228 so that we can always sync the latest updates.

I believe this feature has no blocker anymore, so please go ahead and feel free to let me know if you have any new questions.

Happy Chinese New Year!

[MachengShen]

🎉👍

…

Sent from my iPhone On Feb 13, 2021, at 1:08 PM, Huan (李卓桓) <[email protected]> wrote:  I saw it has been merged already, cheers! Please link the issues together in the future, for example, ref to this issue to jshttp/mime-db#228<jshttp/mime-db#228> so that we can always sync the latest updates. I believe this feature has no blocker anymore, so please go ahead and feel free to let me know if you have any new questions. Happy Chinese New Year! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#19 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALM6ERQGH5UVPKCEZ7IUUXLS6YCMJANCNFSM4W4A243Q>.

[zhangfand]

Please link the issues together in the future, for example, ref to this issue to jshttp/mime-db#228 so that we can always sync the latest updates.

Good call! I wasn't aware this feature exists. Will definitely do that more in the future. I don't foresee any blocker as well. Will address the comments and update the PR.

Happy Lunar New Year to you, too!

[zhangfand]

Hi @huan , I have updated #21 to address comments in this thread. I manually tested the feature by running the added echo.ts with a test OA. Not sure what's the suggested approach to automate testing?