Spawn scanner processes concurrently

[DMRobertson]

The content scanner uses subprocess.run to invoke the scanner script. This is synchronous and does not yield to the event loop. Replace it with the equivalent asyncio call so we can spawn multiple processes and concurrently await their results.

To demonstrate the improvement, I have wired up a rudimentary test script, which downloads 182 avatars from #synapse-dev through the content scanner. Before:

$ git rev-parse HEAD
bbbff949b5ca122a7dbbbdf7799c0dc14fae5f23

$ python perf/scanner_perf.py
number of URLs: 182
......................................................................................................................................................................................
Status codes from scanner server: Counter({200: 174, 403: 4, 502: 4})
Succeeded in 175.86s
......................................................................................................................................................................................
Status codes from scanner server: Counter({200: 178, 403: 4})
Succeeded in 4.26s
Server return code: None

After:

2023-11-28 15:49:57 ✔  $ git rev-parse HEAD
0ba77df7e2056f39fe90d06638e3f39de9a3bfa7

$ python perf/scanner_perf.py
number of URLs: 182
......................................................................................................................................................................................
Status codes from scanner server: Counter({200: 178, 403: 4})
Succeeded in 10.93s
......................................................................................................................................................................................
Status codes from scanner server: Counter({200: 178, 403: 4})
Succeeded in 0.15s
Server return code: None

[DMRobertson]

After:

I was surprised that the second run (with caches warmed up) still benefited from this change. But maybe that's because the Before run hit 502s and ended up retrying them on the second run.

In any case, I think the main result here is the drop from ~3minutes to ~10 seconds.