S
S
Super User2016-02-23 17:56:07
MySQL
Super User, 2016-02-23 17:56:07

Why do errors appear when inserting data into a MySQL table inside a loop?

I am writing a script that pulls links from a page, goes through them with a parser and writes data to MySQL. When one page is parsed, everything is fine, the data is written. As soon as I put the whole thing into a loop, errors start to pour in, like this one:
9e690d905df44ac2b4a025034096749d.PNG
The code itself:

var request = require('sync-request');
var cheerio = require('cheerio');
var mysql = require('mysql');

var connection = mysql.createConnection({
    host: 'localhost',
    user: 'root',
    password: '',
    database: 'prsr'
});

var baseUrl = 'https://en.wikipedia.org';
var startUrl = 'https://en.wikipedia.org/wiki/Category:Video_games_with_3D_graphics';

var res = request('GET', startUrl);
var $ = cheerio.load(res.getBody());

$('#mw-pages').find('.mw-category-group').each(function (index) {    

    var fullUrl = baseUrl + $(this).find('a').attr('href');
    var data = parsePage(fullUrl);

    connection.query('INSERT INTO games SET ?', data, function (err, result) {
        if (err) throw err; // выкидывает ошибку здесь
        console.log(result.insertId);      
    });

});

Answer the question

In order to leave comments, you need to log in

2 answer(s)
V
Vladislav, 2016-02-23
@ghostiam

The error seems to indicate that MySql is terminating the connection due to a timeout. You need to check the MySql settings and increase the maximum connection time.
To set the timeout, there are several options:
connect_timeout
wait_timeout
interactive_timeout

D
Dmitry Voronkov, 2016-02-23
@DmitryVoronkov

Give mysql an error. I am not strong in js. But in my opinion you just have an empty string. You don't check for the existence of the link before writing.
Maybe something like this:

var request = require('sync-request');
var cheerio = require('cheerio');
var mysql = require('mysql');

var connection = mysql.createConnection({
    host: 'localhost',
    user: 'root',
    password: '',
    database: 'prsr'
});

var baseUrl = 'https://en.wikipedia.org';
var startUrl = 'https://en.wikipedia.org/wiki/Category:Video_games_with_3D_graphics';

var res = request('GET', startUrl);
var $ = cheerio.load(res.getBody());

$('#mw-pages').find('.mw-category-group').each(function (index) {    

    var fullUrl = baseUrl + $(this).find('a').attr('href');
    var data = parsePage(fullUrl);
    if (data) {
    connection.query('INSERT INTO games SET ?', data, function (err, result) {
        if (err) throw err; // выкидывает ошибку здесь
        console.log(result.insertId);      
}
    });

});

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question