Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Discussions related to using VirtualBox on Linux hosts.

Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 18. Dec 2019, 15:33

Hello Everybody,

I have a Guest that I need to transfer from host "Host-A" to host "Host-B" (within the same network).

- The Guest is running fine on Host-A, is reachable, can reach everything (it is supposed to).
- The Guest is configured to use a vlan on the brigded eth0 of Host-A.
- Uploading large files with scp to the Guest works like a charm.

But: When that same Guest is running on Host-B:
- uploading large files with scp is extremely slow (when uploading from a Linux machine) or impossible (when uploading from a Windows machine).
- uploading large files with scp works fine when the Guest does *not* use it's vlan configuration

The attached image shows the tcpdump of the failed uploading from windows to Guest.

* The network configuration of Host-A and Host-B are (as far as I can see) identical (/etc/network), /etc/hosts, ...). The environment (router, gateway, dns) is the same.
* The differences between Host-A and Host-B are:

- Host-A is Debian 8 (jessie)
- Host-B is Devuan Ascii 2.1 (~stretch)

- Host-A is using VirtualBox 4.3.36_Debianr105129
- Host-B is using VirtualBox 6.1.0r135406

The guest has the guest additions 6.0 installed (and has no problems when running on Host-A with VirtualBox 4.3)

I made a quick check setting up a devuan ascii 2.1 guest from scratch on Host-B: SCP uploads work just fine until I configure it into the vlan - then the problems start. If I transfer that test-machine to Host-A, everything is fine.

Unfortunately VirtualBox 4.3 cannot be installed on Host-B, nor VirtualBox 6.1 on Host-A, due to unmet dependencies in the different distries (jessie/stretch).
I also cannot upgrade Host-A to stretch just for test, because I need that Guest running and reachable via scp.

Any ideas/hints on what to look for are appreciated.
Nirgants.

=== TCPDUMP ===
Attachments
TcpDump_Guest@Host-B_FromWindows.jpg
TcpDump_Guest@Host-B_FromWindows.jpg (115.92 KiB) Viewed 626 times
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby scottgus1 » 18. Dec 2019, 23:43

We'd probably first need to start here:
Nirgants wrote:Unfortunately VirtualBox 4.3 cannot be installed on Host-B, nor VirtualBox 6.1 on Host-A, due to unmet dependencies in the different distries (jessie/stretch).

Linux distros fork VirtualBox and may change things, so we can't support their forks. If you're using the Linux distro's repositories, you have to either ask on the distro's forum or uninstall the distro's forked VirtualBox and install from https://www.virtualbox.org/wiki/Linux_Downloads. There's also https://www.virtualbox.org/wiki/Download_Old_Builds. Same goes for the Guest Additions, they're forked. Install the GAs using the GAs ISO that comes with official VirtualBox. The GAs should not be downloaded separately.

Chances are you won't want to change the working host A with 4.3.36. But use official 6.1.0 or 6.0.14 for testing host B.

Note the manual's section on networking, and also search the PDF version of the manual, found in the Virtualbox installation folder, for mentions of 'vlan' and variations thereof. Make sure there's nothing in the manual that says they didn't program for what you're trying to do.

If you get this far, then zip and post each guest's .vbox log using the forum's Upload Attachment tab. Also zip and post a guest vbox.log, from cold boot not save-state, to success or failure of the vlan network, to shutdown using the guest OS's shutdown command, to the guest window disappearing.
Human government is like that crazy uncle who hides a quarter in his fist behind his back, then asks you to guess which fist the quarter is in...
No matter which side you choose, Left or Right, both Sides are empty.
scottgus1
Volunteer
 
Posts: 7020
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 19. Dec 2019, 15:33

Thanks for the response!
scottgus1 wrote:Linux distros fork Virualbox and may change things, so we can't support their forks. If you're using the Linux distro's repositories, you have to either ask on the distro's forum or uninstall the distro's forked Virtualbox and install from https://www.virtualbox.org/wiki/Linux_Downloads. There's also https://www.virtualbox.org/wiki/Download_Old_Builds. Same goes for the Guest Additions, they're forked. Install the GAs using the GAs ISO that comes with official Virtualbox. The GAs should not be downloaded separately.

Both installations (Host-A's 4.3 and Host-B's 6.1) were done from using the apt-get sources of virtualbox (deb http://download.virtualbox.org/virtualbox/debian <distri> contrib), but, as said: 4.3 (being too old) cannot be installed on stretch, 6.1 cannot be installed on jessie (that being too old).

Note the manual's section on networking, and also search the PDF version of the manual, found in the Virtualbox installation folder, for mentions of 'vlan' and variations thereof. Make sure there's nothing in the manual that says they didn't program for what you're trying to do.

I did all that, I've been working with virtual box for many years, and in fact the Guest is working perfectly with VirtualBox 4.3.36_Debianr105129 on Debian jessie - but not when moved to VirtualBox 6.1 under Devuan ascii 2.1 (~Debian stretch)

If you get this far, then zip and post each guest's .vbox log using the forum's Upload Attachment tab. Also zip and post a guest vbox.log, from cold boot not save-state, to success or failure of the vlan network, to shutdown using the guest OS's shutdown command, to the guest window disappearing.


I made a quick check setting up a devuan ascii 2.1 guest from scratch on Host-B:
1. SCP uploads work just fine ...
2. ... until I configure it into the vlan - then the problems start.
3. If i configure it back to just bridged everything is fine again.
4. If I move that test-machine (still using vlan) to Host-A, everything is fine.

I have created the logs:
1) Guest on Host-B using vlan (and having scp/ssh problems):
VBox-vlan-B.log
(105.96 KiB) Downloaded 8 times

2) Guest on Host-B without vlan (and being fine):
VBox-no-vlan-B.log
(105.83 KiB) Downloaded 8 times
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby scottgus1 » 19. Dec 2019, 15:44

I'll try to get a look at these logs, have to go to work for now.

Meanwhile, as a possibility, and I'm no Linux guru in any way, I googled "Devuan ascii 2.1" and found it is apparently "a fork of Debian without systemd". They're cutting and chopping Debian in some way. Could they have chopped something important?

Try if possible, on a different hard drive, an install of official Debian 'Stretch' from Debian.org. Put official Virtualbox on it and transfer the guest. See if you get the same results.
Human government is like that crazy uncle who hides a quarter in his fist behind his back, then asks you to guess which fist the quarter is in...
No matter which side you choose, Left or Right, both Sides are empty.
scottgus1
Volunteer
 
Posts: 7020
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby scottgus1 » 19. Dec 2019, 15:45

And see if you can zip and post the working and failing guests' .vbox files.
Human government is like that crazy uncle who hides a quarter in his fist behind his back, then asks you to guess which fist the quarter is in...
No matter which side you choose, Left or Right, both Sides are empty.
scottgus1
Volunteer
 
Posts: 7020
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 19. Dec 2019, 22:39

Thanks again.

scottgus1 wrote:Meanwhile, as a possibility, and I'm no Linux guru in any way, I googled "Devuan ascii 2.1" and found it is apparently "a fork of Debian without systemd". They're cutting and chopping Debian in some way. Could they have chopped something important?

Not really - what they are "chopping and cutting" ;-) is actually concerning the new init system of debian (systemd) back to the originally used init-v. I very much and expertisely doubt effects on the network stack. Actually the no-problem-giving Debian jessie was still running in init-v, too.

Try if possible, on a different hard drive, an install of official Debian 'Stretch' from Debian.org. Put official Virtualbox on it and transfer the guest. See if you get the same results.

A possibility I'll seriously think about - running out of other ideas...

scottgus1 wrote:And see if you can zip and post the working and failing guests' .vbox files.

There are not two vbox files, just one: That of the guest (currently residing on an external drive for easy testing) which has a scp upload-receiving problem when started on Host-B and not when started on Host-A:
guest.vbox.7z
(1.57 KiB) Downloaded 8 times
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby scottgus1 » 20. Dec 2019, 19:50

I don't see anything different in the guest logs, or anything odd in the .vbox file.

You might try getting a log from Host A to see if anything comparable might show.

Though the guest is Bridged to 'eth0', are the network cards labeled 'eth0' on each host the exact same hardware? It could be possible that differences in hardware or host OS drivers could influence how fancier networking schemes perform. The host has to know how to use the networking card for Virtualbox to Bridge to it, so hardware & drivers would influence somewhat.
Human government is like that crazy uncle who hides a quarter in his fist behind his back, then asks you to guess which fist the quarter is in...
No matter which side you choose, Left or Right, both Sides are empty.
scottgus1
Volunteer
 
Posts: 7020
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 30. Dec 2019, 11:30

Hi scottgus1,

sorry for the delay (the season!) and thanks for the help.

I'll check.

Cheers,
Nirgants
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 1. Jan 2020, 22:02

Almost the same cards: The non-working Host-B's is newer, naturally, and the 'Subsystem' is different (marked in red)

Host A (where it is working):
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
Subsystem: Foxconn International, Inc. Device 0d9d

Flags: bus master, fast devsel, latency 0, IRQ 40
I/O ports at e000 [size=256]
Memory at f0004000 (64-bit, prefetchable) [size=4K]
Memory at f0000000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Kernel driver in use: r8169


Host-B (where it is not working):
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller

Flags: bus master, fast devsel, latency 0, IRQ 124
I/O ports at e000 [size=256]
Memory at a1204000 (64-bit, non-prefetchable) [size=4K]
Memory at a1200000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 01
Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
Capabilities: [d0] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
Capabilities: [170] Latency Tolerance Reporting
Kernel driver in use: r8169
Kernel modules: r8169
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby fth0 » 2. Jan 2020, 01:39

Nirgants wrote:The Guest is configured to use a vlan on the brigded eth0 of Host-A.

There are several open bugs (old and new) in the VirtualBox bugtracker regarding the combination of VLAN and the Brigded networking mode. They seem to indicate that the VLAN tags are stripped (by the VirtualBox net filter driver on the host) from packets going from the host to the guest. I don't know wether it ever worked reliably.
fth0
Volunteer
 
Posts: 644
Joined: 14. Feb 2019, 03:06
Location: Germany
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, macOS, ...

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 5. Jan 2020, 13:32

fth0 wrote:
Nirgants wrote:The Guest is configured to use a vlan on the brigded eth0 of Host-A.

There are several open bugs (old and new) in the VirtualBox bugtracker regarding the combination of VLAN and the Brigded networking mode. They seem to indicate that the VLAN tags are stripped (by the VirtualBox net filter driver on the host) from packets going from the host to the guest. I don't know wether it ever worked reliably.


Thanks for the reply.

I found those, too - but, you see: It did and does work reliably for more than a year on Host-A ...
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 5. Jan 2020, 13:40

I have not 'solved' the problem (in the meaning of finding and eliminating the cause) but I have found (thank you, scottgus1, for nudging me in the right direction!) a configuration that does work on Host-B:

Using NIC type virtio

Because of the question by scottgus1 ...

scottgus1 wrote:Though the guest is Bridged to 'eth0', are the network cards labeled 'eth0' on each host the exact same hardware? It could be possible that differences in hardware or host OS drivers could influence how fancier networking schemes perform. The host has to know how to use the networking card for Virtualbox to Bridge to it, so hardware & drivers would influence somewhat.


... I started to check the various nictypes that virtualbox offers (https://www.virtualbox.org/manual/ch06.html#nichardware) and when trying with virtio, the SCP/SFTP/SSH works reliably.

---

PS: Can a moderater please add a [SOLVED] behind the title of this thread (I cannot edit old posts of mine).
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby scottgus1 » 5. Jan 2020, 16:18

Nice! Glad you're up and running.

You should be able to edit your first post's title with the [Solved], which will show in the post list. Look at the upper right corner of each of your posts for the Edit button. Do that on the first post to change the title.
Human government is like that crazy uncle who hides a quarter in his fist behind his back, then asks you to guess which fist the quarter is in...
No matter which side you choose, Left or Right, both Sides are empty.
scottgus1
Volunteer
 
Posts: 7020
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 6. Jan 2020, 14:34

Hi scottgus1,

scottgus1 wrote:You should be able to edit your first post's title with the [Solved], which will show in the post list. Look at the upper right corner of each of your posts for the Edit button. Do that on the first post to change the title.
I can't: All my posts have that edit button, but the first one does not ...

... and anyway, my hooray was premature: uploads to the guest now work fine at e.g. 40Mbyte/s, but downloads from the guest never exceed 170Kbyte/s

Investigating ...

Preliminary Results (same as before, but now only bad for traffic coming from the guest):
scp transfers originating from Host-B to anywhere are full speed of connection (1gbit/s LAN / 10mbit/s WAN)
scp transfers from guest bridged not vlan on Host-B use full speed of connection
scp transfers from guest bridged with vlan on Host-B are as low as 40KB/s and never higher than 170KB/s
scp transfers to the guest use full speed of connection (1gbit/s LAN / 50mbit/s WAN)

It makes no difference from which side the connection was initiated.
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Re: Formerly working SCP/SFTP/SSH not reliably working after transfer to another host

Postby Nirgants » 6. Jan 2020, 20:03

Checked all available nictypes:
Code: Select all   Expand viewCollapse view
AMD PCNet PCI II (Am79C970A)       : Download fast (8 MByte/s)    / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
AMD PCNet FAST III (Am79C973)      : Download fast (8 Mbyte/s)    / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 MT Desktop (82540EM): Download fast (29 Mbyte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 T Server (82543GC)  : Download fast (20 MByte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
Intel PRO/1000 MT Server (82545EM) : Download fast (28 MByte/s)   / Upload aborts after 256KB ("Host is not communicating for 15 seconds")
virtio                             : Download slow (< 170Kbyte/s) / Upload fast (38 Mbyte/s)


The difference between upload/download appears only when configured using vlan - in 'normal' configuration they are (almost) the same (and - as said - also in both cases on Host-A)
Nirgants
 
Posts: 10
Joined: 9. Jan 2017, 10:58
Primary OS: Debian other
VBox Version: OSE Debian
Guest OSses: Linux, Windows

Next

Return to VirtualBox on Linux Hosts

Who is online

Users browsing this forum: No registered users and 21 guests