Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

[Nirgants]

Hello Everybody,

I have a Guest that I need to transfer from host "Host-A" to host "Host-B" (within the same network).

- The Guest is running fine on Host-A, is reachable, can reach everything (it is supposed to).
- The Guest is configured to use a vlan on the brigded eth0 of Host-A.
- Uploading large files with scp to the Guest works like a charm.

But: When that same Guest is running on Host-B:
- uploading large files with scp is extremely slow (when uploading from a Linux machine) or impossible (when uploading from a Windows machine).
- uploading large files with scp works fine when the Guest does *not* use it's vlan configuration

The attached image shows the tcpdump of the failed uploading from windows to Guest.

* The network configuration of Host-A and Host-B are (as far as I can see) identical (/etc/network), /etc/hosts, ...). The environment (router, gateway, dns) is the same.
* The differences between Host-A and Host-B are:

- Host-A is Debian 8 (jessie)
- Host-B is Devuan Ascii 2.1 (~stretch)

- Host-A is using VirtualBox 4.3.36_Debianr105129
- Host-B is using VirtualBox 6.1.0r135406

The guest has the guest additions 6.0 installed (and has no problems when running on Host-A with VirtualBox 4.3)

I made a quick check setting up a devuan ascii 2.1 guest from scratch on Host-B: SCP uploads work just fine until I configure it into the vlan - then the problems start. If I transfer that test-machine to Host-A, everything is fine.

Unfortunately VirtualBox 4.3 cannot be installed on Host-B, nor VirtualBox 6.1 on Host-A, due to unmet dependencies in the different distries (jessie/stretch).
I also cannot upgrade Host-A to stretch just for test, because I need that Guest running and reachable via scp.

Any ideas/hints on what to look for are appreciated.
Nirgants.

=== TCPDUMP ===

[scottgus1]

We'd probably first need to start here:

Unfortunately VirtualBox 4.3 cannot be installed on Host-B, nor VirtualBox 6.1 on Host-A, due to unmet dependencies in the different distries (jessie/stretch).

Linux distros fork VirtualBox and may change things, so we can't support their forks. If you're using the Linux distro's repositories, you have to either ask on the distro's forum or uninstall the distro's forked VirtualBox and install from https://www.virtualbox.org/wiki/Linux_Downloads. There's also https://www.virtualbox.org/wiki/Download_Old_Builds. Same goes for the Guest Additions, they're forked. Install the GAs using the GAs ISO that comes with official VirtualBox. The GAs should not be downloaded separately.

Chances are you won't want to change the working host A with 4.3.36. But use official 6.1.0 or 6.0.14 for testing host B.

Note the manual's section on networking, and also search the PDF version of the manual, found in the Virtualbox installation folder, for mentions of 'vlan' and variations thereof. Make sure there's nothing in the manual that says they didn't program for what you're trying to do.

If you get this far, then zip and post each guest's .vbox log using the forum's Upload Attachment tab. Also zip and post a guest vbox.log, from cold boot not save-state, to success or failure of the vlan network, to shutdown using the guest OS's shutdown command, to the guest window disappearing.

[Nirgants]

Thanks for the response!

Linux distros fork Virualbox and may change things, so we can't support their forks. If you're using the Linux distro's repositories, you have to either ask on the distro's forum or uninstall the distro's forked Virtualbox and install from https://www.virtualbox.org/wiki/Linux_Downloads. There's also https://www.virtualbox.org/wiki/Download_Old_Builds. Same goes for the Guest Additions, they're forked. Install the GAs using the GAs ISO that comes with official Virtualbox. The GAs should not be downloaded separately.

Both installations (Host-A's 4.3 and Host-B's 6.1) were done from using the apt-get sources of virtualbox (deb http://download.virtualbox.org/virtualbox/debian <distri> contrib), but, as said: 4.3 (being too old) cannot be installed on stretch, 6.1 cannot be installed on jessie (that being too old).

Note the manual's section on networking, and also search the PDF version of the manual, found in the Virtualbox installation folder, for mentions of 'vlan' and variations thereof. Make sure there's nothing in the manual that says they didn't program for what you're trying to do.

I did all that, I've been working with virtual box for many years, and in fact the Guest is working perfectly with VirtualBox 4.3.36_Debianr105129 on Debian jessie - but not when moved to VirtualBox 6.1 under Devuan ascii 2.1 (~Debian stretch)

If you get this far, then zip and post each guest's .vbox log using the forum's Upload Attachment tab. Also zip and post a guest vbox.log, from cold boot not save-state, to success or failure of the vlan network, to shutdown using the guest OS's shutdown command, to the guest window disappearing.

I made a quick check setting up a devuan ascii 2.1 guest from scratch on Host-B:
1. SCP uploads work just fine ...
2. ... until I configure it into the vlan - then the problems start.
3. If i configure it back to just bridged everything is fine again.
4. If I move that test-machine (still using vlan) to Host-A, everything is fine.

I have created the logs:
1) Guest on Host-B using vlan (and having scp/ssh problems):

VBox-vlan-B.log

(105.96 KiB) Downloaded 13 times

2) Guest on Host-B without vlan (and being fine):

VBox-no-vlan-B.log

(105.83 KiB) Downloaded 13 times

[scottgus1]

I'll try to get a look at these logs, have to go to work for now.

Meanwhile, as a possibility, and I'm no Linux guru in any way, I googled "Devuan ascii 2.1" and found it is apparently "a fork of Debian without systemd". They're cutting and chopping Debian in some way. Could they have chopped something important?

Try if possible, on a different hard drive, an install of official Debian 'Stretch' from Debian.org. Put official Virtualbox on it and transfer the guest. See if you get the same results.

[scottgus1]

And see if you can zip and post the working and failing guests' .vbox files.

[Nirgants]

Thanks again.

Meanwhile, as a possibility, and I'm no Linux guru in any way, I googled "Devuan ascii 2.1" and found it is apparently "a fork of Debian without systemd". They're cutting and chopping Debian in some way. Could they have chopped something important?

Not really - what they are "chopping and cutting" is actually concerning the new init system of debian (systemd) back to the originally used init-v. I very much and expertisely doubt effects on the network stack. Actually the no-problem-giving Debian jessie was still running in init-v, too.

Try if possible, on a different hard drive, an install of official Debian 'Stretch' from Debian.org. Put official Virtualbox on it and transfer the guest. See if you get the same results.

A possibility I'll seriously think about - running out of other ideas...

And see if you can zip and post the working and failing guests' .vbox files.

There are not two vbox files, just one: That of the guest (currently residing on an external drive for easy testing) which has a scp upload-receiving problem when started on Host-B and not when started on Host-A:

guest.vbox.7z

(1.57 KiB) Downloaded 13 times

[scottgus1]

I don't see anything different in the guest logs, or anything odd in the .vbox file.

You might try getting a log from Host A to see if anything comparable might show.

Though the guest is Bridged to 'eth0', are the network cards labeled 'eth0' on each host the exact same hardware? It could be possible that differences in hardware or host OS drivers could influence how fancier networking schemes perform. The host has to know how to use the networking card for Virtualbox to Bridge to it, so hardware & drivers would influence somewhat.

[Nirgants]

Hi scottgus1,

sorry for the delay (the season!) and thanks for the help.

I'll check.

Cheers,
Nirgants

[Nirgants]

Almost the same cards: The non-working Host-B's is newer, naturally, and the 'Subsystem' is different (marked in red)

Host A (where it is working):

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
Subsystem: Foxconn International, Inc. Device 0d9d
Flags: bus master, fast devsel, latency 0, IRQ 40
I/O ports at e000
Memory at f0004000 (64-bit, prefetchable) [size=4K]
Memory at f0000000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Kernel driver in use: r8169

Host-B (where it is not working):

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
Flags: bus master, fast devsel, latency 0, IRQ 124
I/O ports at e000
Memory at a1204000 (64-bit, non-prefetchable) [size=4K]
Memory at a1200000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Capabilities: [170] Latency Tolerance Reporting
Kernel driver in use: r8169
Kernel modules: r8169

[fth0]

The Guest is configured to use a vlan on the brigded eth0 of Host-A.

There are several open bugs (old and new) in the VirtualBox bugtracker regarding the combination of VLAN and the Brigded networking mode. They seem to indicate that the VLAN tags are stripped (by the VirtualBox net filter driver on the host) from packets going from the host to the guest. I don't know wether it ever worked reliably.

[Nirgants]

The Guest is configured to use a vlan on the brigded eth0 of Host-A.

There are several open bugs (old and new) in the VirtualBox bugtracker regarding the combination of VLAN and the Brigded networking mode. They seem to indicate that the VLAN tags are stripped (by the VirtualBox net filter driver on the host) from packets going from the host to the guest. I don't know wether it ever worked reliably.

Thanks for the reply.

I found those, too - but, you see: It did and does work reliably for more than a year on Host-A ...

[Nirgants]

I have not 'solved' the problem (in the meaning of finding and eliminating the cause) but I have found (thank you, scottgus1, for nudging me in the right direction!) a configuration that does work on Host-B:

Using NIC type virtio

Because of the question by scottgus1 ...

Though the guest is Bridged to 'eth0', are the network cards labeled 'eth0' on each host the exact same hardware? It could be possible that differences in hardware or host OS drivers could influence how fancier networking schemes perform. The host has to know how to use the networking card for Virtualbox to Bridge to it, so hardware & drivers would influence somewhat.

... I started to check the various nictypes that virtualbox offers (https://www.virtualbox.org/manual/ch06.html#nichardware) and when trying with virtio, the SCP/SFTP/SSH works reliably.

---

PS: Can a moderater please add a [SOLVED] behind the title of this thread (I cannot edit old posts of mine).

[scottgus1]

Nice! Glad you're up and running.

You should be able to edit your first post's title with the [Solved], which will show in the post list. Look at the upper right corner of each of your posts for the Edit button. Do that on the first post to change the title.

[Nirgants]

Hi scottgus1,

You should be able to edit your first post's title with the [Solved], which will show in the post list. Look at the upper right corner of each of your posts for the Edit button. Do that on the first post to change the title.

I can't: All my posts have that edit button, but the first one does not ...

... and anyway, my hooray was premature: uploads to the guest now work fine at e.g. 40Mbyte/s, but downloads from the guest never exceed 170Kbyte/s

Investigating ...

Preliminary Results (same as before, but now only bad for traffic coming from the guest):
scp transfers originating from Host-B to anywhere are full speed of connection (1gbit/s LAN / 10mbit/s WAN)
scp transfers from guest bridged not vlan on Host-B use full speed of connection
scp transfers from guest bridged with vlan on Host-B are as low as 40KB/s and never higher than 170KB/s
scp transfers to the guest use full speed of connection (1gbit/s LAN / 50mbit/s WAN)

It makes no difference from which side the connection was initiated.

[Nirgants]

Checked all available nictypes:

Code: Select all

AMD PCNet PCI II (Am79C970A)       : Download fast (8 MByte/s)    / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
AMD PCNet FAST III (Am79C973)      : Download fast (8 Mbyte/s)    / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 MT Desktop (82540EM): Download fast (29 Mbyte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 T Server (82543GC)  : Download fast (20 MByte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 MT Server (82545EM) : Download fast (28 MByte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
virtio                             : Download slow (< 170Kbyte/s) / Upload fast (38 Mbyte/s)

The difference between upload/download appears only when configured using vlan - in 'normal' configuration they are (almost) the same (and - as said - also in both cases on Host-A)