Testing Network Applications in Python

Question

Python

Case 1.
In my Python application, I perform a "simple" action on a specific site or remote computer. For example, ping or getting the HTML content of a site. And I want to test this method in a Unit test. But testing on a real site is not an option for obvious reasons.

Question: how in such cases is a wrapper made around my script, which intercepts network requests and which I can customize as I wish?

Case 2.
In my Python application, I want to perform a "complex" action with a specific site. For example, cleverly parse its contents. But there is a fear that if I fiddle with the site for a long time, debug my requests “live”, the site administrator can cut through this and ban me.

Question: Are there any automation tools that make the most similar copy of a website on my computer? Or is it done manually in such cases using some kind of web framework?

Case 3.
I want to test a self-written client-server application on a more or less live network, albeit an emulated one. My first thought: take a virtual machine, create N machines, install operating systems there... But it's heavy and requires a lot of memory.
Maybe there is a lightweight program that implements only the networking stack "as in adult OSes" and the Python interpreter?

Answer 1

1. you can raise the web server in a separate thread or process, this web server can emulate the operation of the site. There are a lot of specific options for how to do this: threading, multiprocessing, using libraries like gevent or twisted
2. There are a lot of programs that allow you to make a "copy" of the site. the simplest option is probably wget, it can pull out almost the entire site with one command, it can follow links. then the web server rises (perhaps one of the options mentioned in 1) and you can test on this miracle
3. if we talk about virtual machines, then this is quite a working option, for testing python applications, 256 mb per machine should be enough, 4 machines is a gigabyte of RAM, which is quite lifting for any car with 4 GB of memory. if you want something simpler, then nothing prevents, as already mentioned, just connecting to the localhost, programs are launched either in different processes, or even in different threads. to test network drops, delays, disconnects and other things, you can use self-written tcp proxies (we do this), proxies can be controlled in very different ways, emulating a variety of “sad” situations
Shl. several virtual machines can be easily raised using vagrant, and you can install the necessary software on them using puppet, chef, salt - according to your taste

Answer 2

For the second case, caching responses by URL key suits me, maybe it will suit you.

Answer 3

It seems to me that there is little information and too vague, so the answers are as follows:
1. You can make low-level requests separately from high-level work and simply replace the low-level module or configure it, and the top level with unit tests will not change.
2. You can cache as advised above. If something is simple, then I simply replaced and read local files, which I previously pulled from the site for certain of my requests.
3. I didn't really understand why virtual machines. If you write python scripts and each will be executed independently, then machines are not needed. Everyone will have their own cookies, environment, their own configs, and so on. Just run in parallel.

Answer 4

Python has very handy libraries for patching functions and attributes on the fly for tests. We are using flexmock. With its help, for example, you can replace the functions of the urllib library (or any other), then your function will not make real requests to the network, but will return what you need.
There is a presentation here, see from the 47th slide pycon.ru/program/content/test-driven-development/

Answer 5

What complex answers) If the question is still relevant, then the easiest way to write unit tests in isolation from external data sources is to use Mock frameworks (for example, mox).
The 3rd question is also solved without the invention of bicycles by utilities for testing the load. The simplest example of such a utility is Apache Bench. I know that in nature there are more complex solutions that you can look for yourself.