W
W
WhiteNinja2016-10-14 11:23:32
C++ / C#
WhiteNinja, 2016-10-14 11:23:32

How to implement queue processing service in C# + EF 6?

Good day!
Technologies:
ASP.NET MVC 5, Entity Framework 6, SQL Server 2008/MySQL
Task:
There is a web application (ASP.NET MVC) in which the user uploads files (upload) with a certain structure (one file at a time). The files contain a lot of data (medium-structured). After uploading the file, it is necessary to "scatter" all the data from the uploaded file into about 15 tables using certain business rules.
Idea:
Since working out business rules and "scattering" data from one file can take a lot of time, it was decided to organize a certain queue "for unpacking" (by implementing the queue using the QueueTable table described below), and process the queue itself by a separate service (or windows service, or a simple console application) on the server.
It turns out after upload'a from ASP.NET MVC, the data about the file is stored in a table like this with the status InProgress (1):

CREATE TABLE QueueTable
(
  Id int,
  FilePath varchar(255),
  StatusId int
);

StatusId takes the values ​​InProgress(1), Errors(2), Success(3).
Accordingly, the console application must pull out all rows with the InProgress (1) status from this queue, work out all business rules and "scatter" data across all necessary tables, and if everything went well, change the status in the queue to Success (3) or if it crashed Exception, then change the status to Errors(2).
Questions:
1. Is the approach to solving the problem successfully chosen, or is there a more optimal solution for "complex" file processing?
2. I would also like to parallelize the work of processing the queue. Can you please tell me an example of how such a service (console application) could look like in order to understand which way to look? I would also like to use the Entity framework in this parallelized service when working with the database, if possible.
Thanks in advance for any help!

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vitaly Pukhov, 2016-10-14
@WhiteNinja

there is an application with approximately the same architecture, only the “file” itself, in my case, I put it in the same place in the database so that the service does not depend on any folder. There are no questions about reliability and rate of fire at all, everything works very quickly. Parallel processing was done in the following way:
Thus, it turns out that the threads "select" a task for themselves, process it and update the status, and so on in a circle.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question