How to properly organize VGA-passthrought in qemu-kvm?

Question

linux

I'm trying to set up forwarding a Radeon R7 240 video card to a virtual machine (windows7).
When using ATI Open Source (radeon) I encountered a problem that pci-stub does not block the card and when trying to do:

echo 0000:06:00.0 > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind

The system "freezes": i.e. one gets the impression that both cards completely fall off, while dmes (kern.log) are clean.
If you use fglx of the pci-stub driver normally blocks the device

dmesg | grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-25-generic root=UUID=0e0cb9d5-fbcf-4a95-88cd-41219cd6c3e2 ro amd_iommu=on max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush pci-stub.ids=1002:6613,1002:aab0 quiet splash modeset=1 vt.handoff=7
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-25-generic root=UUID=0e0cb9d5-fbcf-4a95-88cd-41219cd6c3e2 ro amd_iommu=on max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush pci-stub.ids=1002:6613,1002:aab0 quiet splash modeset=1 vt.handoff=7
[    3.761629] pci-stub: add 1002:6613 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.761645] pci-stub 0000:06:00.0: claimed by stub
[    3.761652] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.761663] pci-stub 0000:06:00.1: claimed by stub
[    4.902238] pci-stub 0000:06:00.0: enabling device (0000 -> 0003)

However, vfio cannot remap for this map:

qemu-system-x86_64: -device vfio-pci,host=06:00.0: vfio: Error: Failed to setup INTx fd: Device or resource busy
qemu-system-x86_64: -device vfio-pci,host=06:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=06:00.0: Device 'vfio-pci' could not be initialized

dmesg | tail | grep vfio
[  435.580406] vfio_ecap_init: 0000:06:00.0 hiding ecap [email protected]
[  435.586186] genirq: Flags mismatch irq 45. 00000080 (vfio-intx(0000:06:00.0)) vs. 00000820 (fglrx[0]@PCI:6:0:0)

Tell me what I'm doing wrong and where to dig? Can I somehow force pci-stub to block the card? Or make a remap for the map with your hands?
Some information about the system:

lspci -nn | grep Radeon
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] [1002:68ba]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series] [1002:aa58]
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] [1002:6613]
06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

dmesg | grep  IOMMU
[    1.246790] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    4.547794] AMD IOMMUv2 driver by Joerg Roedel <[email protected]>
[    4.547797] AMD IOMMUv2 functionality not available on this system
[    4.902081] <6>[fglrx] IOMMU is enabled, CrossFire are not supported on this platform
[    4.902082] <6>[fglrx] Disable IOMMU in BIOS options or kernel boot parameters to support CF

cat /etc/default/grub
GRUB_CMDLINE_LINUX="amd_iommu=on max_loop=64 iommu=pt iommu=1 amd_iommu=fullflush pci-stub.ids=1002:6613,1002:aab0"

PS: if you forward the HD 6770, then everything works fine (I didn’t try it with the radeon module, fglx), but it’s interesting for me to forward the R7 240.

Answer 1

After updating the kernel to:

uname -r
4.1.0-040100rc2-generic

My situation has not changed: drm still loaded radeon, which initialized the video device and did not allow pci-stub / vfio-pci to connect the device.
Manual shutdown of the device:
Crashed the system (kernel BUG).
In an attempt to solve the problem, I did the following:
1. Registered device IDs for vfio-pci

cat /etc/modprobe.d/vfio-pci.conf 
options vfio-pci ids=1002:6613,1002:aab0

2. Added softdep and install options for drm and radeon modules

cat /etc/modprobe.d/radeon.conf 
softdep radeon pre: vfio vfio_pci
install radeon /sbin/modprobe vfio; /sbin/modprobe vfio_pci
cat /etc/modprobe.d/drm.conf 
softdep drm pre: vfio vfio_pci post: radeon
softdep drm_kms_helper pre: vfio vfio_pci post: radeon
install drm /sbin/modprobe vfio; /sbin/modprobe vfio_pci

However, that didn't help either. My meager knowledge of the linux boot process turned out to be powerless, but I can assume that this is due to the fact that drm / radeon are loaded directly by the kernel (not through modprobe), and therefore they are loaded before the softdep and install options are processed.
Solution (crutch): let me remind you once again that I have 2 cards - HD 6770 (Juniper) and R7 240 (Oland). Because I do not expect to use R7 240 (Oland) in the host system, so I decided to cut it out completely. Therefore, I removed all firmware OLAND*.bin from /lib/firmware/radeon/.

rm -f /lib/firmware/radeon/OLAND*.bin
update-initramfs -u
init 6

When updating the initramfs, I received a bunch of warnings that no firmware was found for OLAND. After the reboot, I saw the long-awaited:

06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240] [1002:6613] (prog-if 00 [VGA controller])
  Subsystem: ASUSTeK Computer Inc. Device [1043:04c5]
  Flags: bus master, fast devsel, latency 0, IRQ 46
  Memory at b0000000 (64-bit, prefetchable) [size=256M]
  Memory at fe700000 (64-bit, non-prefetchable) [size=256K]
  I/O ports at c000 [size=256]
  Expansion ROM at fe740000 [disabled] [size=128K]
  Capabilities: <access denied>
  Kernel driver in use: vfio-pci

06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
  Subsystem: ASUSTeK Computer Inc. Device [1043:aab0]
  Flags: bus master, fast devsel, latency 0, IRQ 21
  Memory at fe760000 (64-bit, non-prefetchable) [size=16K]
  Capabilities: <access denied>
  Kernel driver in use: vfio-pci

The guest Windows 7 loaded remarkably and rated the graphics adapter 6.8 (for HD 6770 - 7.3)
I understand that this is a "crutch" and probably should not be done this way. If you have a more elegant solution, please let me know.

Answer 2

1) Proprietary drivers cannot be used on the host, only in the guest.
2) In the bios of the card, do not forget to specify which video card to use as the primary one?
3) xorg.cfg removed?
4) What does "suspend" mean?