UPDATE : All is resolved, and running perfectly. Smooth sailing!
———
Original Post
For those of you that appreciate a little Geek-speak, there are 3 distinct issues causing the current agent socket server crash, which in turn is affecting a small % of users by kicking them out of the dashboard intermittently :
1. mqtt reading of closed socket
2. parsing of object id in mongodb driver
3. websocket parsing
We have fixed issue no.1 and it has been deployed to production.
There are 2 distinct issues causing the visitor socket server crash
1. parsing of object id in mongodb driver
2. websocket parsing
The intermittent dashboard outages caused by agent socket server keep restarting, and putting the agent application servers under more load that they can handle.
We have temporarily started additional agent application servers to keep up with the agent socket restarts, though currently working on fixing object id parsing issue and websocket parsing issue.
We’ll update this post once all is back to normal (minutes or hours, not days!).
Thanks again for your patience, we truly do apologize for any inconvenience this has caused you or your business – we know how much you all rely on the stability of our service.
Today was not our finest of days!
