[sally555]

И отправлять не число, а ArrayBuffer.
https://developer.mozilla.org/en-US/.../API/WebSocket

Спасибо за ссылку, разобрался, все заработало.
Вот примерчик для наглядности нацарапал, массивы с разной разрядностью 8,16 и 32:

function sendbin(){
//var ArrBuffDt = new ArrayBuffer(10);
var ArrBuffData = new Int8Array(10);
ArrBuffData[0]=17;
ArrBuffData[1]=12;
ArrBuffData[2]=2;
ArrBuffData[3]=0;
ArrBuffData[4]=1;
ArrBuffData[5]=5;
ArrBuffData[6]=12;
ArrBuffData[7]=17;
ArrBuffData[8]=100;
ArrBuffData[9]=20;
soc.send(ArrBuffData);
//delete ArrBuffDt;
delete ArrBuffData;

var ArrBuffDt = new ArrayBuffer(20);
var ArrBuffData = new Int16Array(ArrBuffDt);
ArrBuffData[0]=17;
ArrBuffData[1]=12;
ArrBuffData[2]=2;
ArrBuffData[3]=0;
ArrBuffData[4]=1;
ArrBuffData[5]=5;
ArrBuffData[6]=12;
ArrBuffData[7]=17;
ArrBuffData[8]=100;
ArrBuffData[9]=20;
soc.send(ArrBuffDt);
delete ArrBuffDt;
delete ArrBuffData;

var ArrBuffDt = new ArrayBuffer(40);
var ArrBuffData = new Int32Array(ArrBuffDt);
ArrBuffData[0]=17;
ArrBuffData[1]=12;
ArrBuffData[2]=2;
ArrBuffData[3]=0;
ArrBuffData[4]=1;
ArrBuffData[5]=5;
ArrBuffData[6]=12;
ArrBuffData[7]=17;
ArrBuffData[8]=100;
ArrBuffData[9]=20;
soc.send(ArrBuffData);
delete ArrBuffDt;
delete ArrBuffData;
}

А это то что приходит на стороне сервера:

[1] Disconnected from ESP8266!
2
[0] Connected from 192.168.4.2 url: /
4
[0] get binary lenght: 10

[HEXDUMP] Address: 0x3FFF30B8 len: 0xA (10)
[0x3FFF30B8] 0x00000000: 11 0C 02 00 01 05 0C 11 64 14 
4
[0] get binary lenght: 20

[HEXDUMP] Address: 0x3FFF36F8 len: 0x14 (20)
[0x3FFF36F8] 0x00000000: 11 00 0C 00 02 00 00 00 01 00 05 00 0C 00 11 00 
[0x3FFF3708] 0x00000010: 64 00 14 00 
4
[0] get binary lenght: 40

[HEXDUMP] Address: 0x3FFF3F50 len: 0x28 (40)
[0x3FFF3F50] 0x00000000: 11 00 00 00 0C 00 00 00 02 00 00 00 00 00 00 00 
[0x3FFF3F60] 0x00000010: 01 00 00 00 05 00 00 00 0C 00 00 00 11 00 00 00 
[0x3FFF3F70] 0x00000020: 64 00 00 00 14 00 00 00 
1
[0] Disconnected from ESP8266!