I ask you to suggest Test Framework on the principle of a black box for testing console applications

Question

Software testing

I need a tool that would allow me to organize testing of console applications.

The main criteria are:

1. multiplatform
2. Black box testing
3. Minimalism

The test plan that this test system understands should be something like this:
[general]
[general]
testobject = ..\..\my_tools.exe
expected_reports = ..\expected_reports\
output_logs = ..\output_logs\
test_vectors = ..\test_vectors\
output_files = ..\output\
out_postfix = output

[setup_emptyfile]
input_file = sample.bin
input_md5 = 104B0FF4DA7DA7B91C447830BEECBABD
cmdline = --source_file %input_file% --output_file %input_file%_%out_postfix%.bin --nologo

[test_emptyfile]
exit_code = 5
outlog_md5 = 221CCDCAB9DF2A06150AA93D07CA455A
output_md5 = D41D8CD98F00B204E9800998ECF8427E
stdout_md5 = 8E5338AD169E2D7DDB57109BB2A5FAB2
stderr_md5 = 8E5338AD169E2D7DDB57109BB2A5FAB2

[teardown_emptyfile]
output_file = sample_output.bin
stdout_file = sample_stdout.bin
stderr_file = sample_stderr.bin



Answer 1

Quite directly in the form that you want to suggest I can not. But there is an Aruba plugin for the wonderful cucumber BDD framework .
You can see how its use looks, for example, in the tests (features) of the cucumber itself . An example of a specific feature .
You can understand how the steps in features are arranged by reading the descriptions of these steps .
You can even write scripts in Russian .
In general, good visual documentation is obtained from cucumber tests.
An example of documentation automatically generated from cucumber tests - in fact, one of the types of documentation for cucumber or documentation for Rspec2. For example, here is the documentation for the feature I linked above.
Aruba: github.com/cucumber/aruba
Aruba fork: github.com/byrnejb/aruba
Take your pick .

Answer 2

I considered LoadRunner and cocumber, but dismissed them due to violation of criterion #3 (see my topic). For example, for a tool like readelf (famous among UNIX users), which I am currently developing, a solution based on LoadRunner and cocumber will be like firing cannons at sparrows.
For now, I'm leaning towards my own development in Python 3.2 due to the multiplatform nature and wide popularity of the latter.
I want to test tools like readelf, as well as tools that process the input stream from STDIN or a file and give the result to STDOUT or to a file. At the same time, I conclude an agreement with myself that the tool does not have the right to write error messages to STDOUT and writes them only to STDERR or to a log file.

Answer 3

It's been quite a while of my searching. In general, an open-source project code.google.com/p/non-smoker/ was launched to solve its problem . Probably, somewhere there is a project that meets my requirements, but since I haven’t found it to this day, it makes no sense to wait and leave for later those testing tasks that I want to solve for a long time. I welcome any criticism or suggestions.