How to correctly calculate the effectiveness of an online consultant?

Question

Conversion

Faced such a problem.
Installed an online consultant in the online store. Everything works, the manager answers, everything is fine. But then we thought about how to analyze the effectiveness of the chat, how it affected the conversion and came to a standstill.
All sellers of such services in their descriptions claim that an online consultant increases conversion, increases efficiency, etc. But there is no information anywhere on how to calculate this efficiency. I found only information on how to calculate the hours of work of a manager in a consultant, the time to answer questions, etc. But how to calculate the benefit of a chat for a store is not clear. Plus, there are tools for how to set up metrics, integrate with CRM, etc. But it’s not clear how to adequately evaluate statistics.
Let's say we set a composite goal in the metric: chat message + checkout. Those. now we can see how many people have written to the chat and how many people have left the order. But this does not say much, because. those people who wrote and left the order, would they leave the order if there was no chat?
Let's say 5,000 people came to the site in a day. The order was placed by 50 people, i.е. conversion 1%. Suppose that out of 5,000 visitors, 10 wrote to the chat. And out of these 10, 2 people placed an order. Those. those who wrote in the chat had a 20% conversion rate. It turns out that 4990 people did not write to the chat and left 48 orders, i.e. conversion 0.96%. Is this the correct line of reasoning? Of the five thousand people, 10 wrote to the chat and left 2 orders for ten, and the remaining 4990 people left only 48 orders. Those. the conclusion is that those who wrote in the chat place an order more willingly.
Or you can’t take it so straightforwardly and you need to take into account a bunch of other factors. Well, for example, we can assume that those people who are charged to leave an order, they are just more willing to write to the chat. Those. it was not the chat that caused the order to be placed, but rather, the person was already ready to leave the order and therefore he needed a chat.
Can someone give advice on this topic or can suggest some articles?

Answer 1

It is logical to compare with historical data. There were so many non-zero people who bought so many. There were so
many people, such and such a percentage writes to the chat, they buy so many.

Answer 2

I subscribe to the above. Without historical data, all calculations are in the furnace.
IMHO, you're overreacting. It is impossible to accurately assess the importance of an online consultant on the path to conversion. The consultant may be in a bad mood, for example, and he did not communicate well)) Or people talked, left, and a week later returned and bought, because the consultant was persuasive and responsive, but you do not track it.
Plus, counting the number of conversions is not entirely correct.
For example, consultants can increase the average check. Those. a person wanted to buy something specific, talked to a consultant, he told about a similar product, but with improved functionality - a person bought more expensive. Those. must be analyzed from all angles.

Answer 3

Also compare the check when ordering with chat and without chat. And also - compare the cost of the chat and the person in it with the profit from it.
But something tells me that a conversion of 0.2% of visitors to chat means that chat is simply not needed.