Server responds with empty packet during session negotiation resulting in client giving a malformed packet error

I'm attempting to connect to a remote mysql server. This happens 100% of the time.

client: mysql Ver 14.14 Distrib 5.7.12, for Win32 (AMD64)

server: 5.0.95

This is the error I get:

C:\>mysql -h example.com -P 3306 -D prod_rcadb -u username -p
Enter password: **********
ERROR 2027 (HY000): Malformed packet

same error with mysqladmin:

C:\>mysqladmin -h example.com -P 3306 -u username -p version
Enter password: **********
mysqladmin: connect to server at '10.106.24.79' failed
error: 'Malformed packet'

So I took a pcap, just to get an idea of what that conversation looked like.

TCP handshake:

1              2016-04-14 11:18:48.910690         0.000000              137.69.150.80                     10.106.24.79       TCP        66                51157→3306 [SYN] Seq=0 Win=8192 Len=0 MSS=1428 WS=256 SACK_PERM=1   8192
2              2016-04-14 11:18:49.019893         0.109203              10.106.24.79                       137.69.150.80     TCP        66                3306→51157 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=256           5840
3              2016-04-14 11:18:49.019893         0.000000              137.69.150.80                     10.106.24.79       TCP        54                51157→3306 [ACK] Seq=1 Ack=1 Win=65536 Len=0          256

Mysql connection negotiation:

```
4 2016-04-14 11:18:49.144696 0.124803 10.106.24.79 137.69.150.80 MySQL 110 Server Greeting proto=10 version=5.0.95 23
5 2016-04-14 11:18:49.144696 0.000000 137.69.150.80 10.106.24.79 MySQL 119 Login Request user=bigdata 256
6 2016-04-14 11:18:49.144696 0.000000 10.106.24.79 137.69.150.80 TCP 60 3306→51157 [ACK] Seq=57 Ack=66 Win=5888 Len=0 23
7 2016-04-14 11:18:49.316301 0.171605

The packet that I thought was empty was actually not, it was an Old Auth Switch Request:

Payload
1     [fe]
Fields
status (1) -- 0xfe
Returns
Protocol::AuthSwitchResponse with old password hash
Example
01 00 00 02 fe

The 1, 2, 254 that wireshark parsed is actually 01 00 00 02 fe if you look at the actual byte strings.

So it isn't that there is a misunderstanding, the server understands completely and correctly responds and the client correctly terminates because it cannot negotiate down. The protocol is not too old as it was changed in 4.1, so both 5.0 and 5.7 understand each other perfectly. This should be a clearer error message.

The client is too new to use --skip-secure-auth (reference). They purposefully removed the ability to negotiate down prior to 5.7.

So my options are to allow new passwords on the server (which is a user specific configuration, not global server option) or to use an older binary.

The specific configuration issue is based on the username I was given. At some point in the past, someone using this username was using an older client and they changed the password method:

B.5.2.4 Client does not support authentication protocol

The current implementation of the authentication protocol uses a password hashing algorithm that is incompatible with that used by older (pre-4.1) clients. Attempts to connect to a 4.1 or newer server with an older client may fail with the following message:

shell> mysql
Client does not support authentication protocol requested
by server; consider upgrading MySQL client

To deal with this problem, the preferred solution is to upgrade all client programs to use a 4.1.1 or newer client library. If that is not possible, use one of the following approaches:

    To connect to the server with a pre-4.1 client program, use an account that still has a pre-4.1-style password.

    Reset the password to pre-4.1 style for each user that needs to use a pre-4.1 client program. This can be done using the SET PASSWORD statement and the OLD_PASSWORD() function. As of MySQL 5.6.6, it is also necessary to first ensure that the authentication plugin for the account is mysql_old_password:

    mysql> UPDATE mysql.user SET plugin = 'mysql_old_password'
    mysql> WHERE User = 'some_user' AND Host = 'some_host';
    mysql> FLUSH PRIVILEGES;
    mysql> SET PASSWORD FOR
        -> 'some_user'@'some_host' = OLD_PASSWORD('new_password');

StackExchange Database Administrators Q#135343, answer score: 13