Answer the question
In order to leave comments, you need to log in
Why do errors appear when inserting data into a MySQL table inside a loop?
I am writing a script that pulls links from a page, goes through them with a parser and writes data to MySQL. When one page is parsed, everything is fine, the data is written. As soon as I put the whole thing into a loop, errors start to pour in, like this one:
The code itself:
var request = require('sync-request');
var cheerio = require('cheerio');
var mysql = require('mysql');
var connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: '',
database: 'prsr'
});
var baseUrl = 'https://en.wikipedia.org';
var startUrl = 'https://en.wikipedia.org/wiki/Category:Video_games_with_3D_graphics';
var res = request('GET', startUrl);
var $ = cheerio.load(res.getBody());
$('#mw-pages').find('.mw-category-group').each(function (index) {
var fullUrl = baseUrl + $(this).find('a').attr('href');
var data = parsePage(fullUrl);
connection.query('INSERT INTO games SET ?', data, function (err, result) {
if (err) throw err; // выкидывает ошибку здесь
console.log(result.insertId);
});
});
Answer the question
In order to leave comments, you need to log in
The error seems to indicate that MySql is terminating the connection due to a timeout. You need to check the MySql settings and increase the maximum connection time.
To set the timeout, there are several options:
connect_timeout
wait_timeout
interactive_timeout
Give mysql an error. I am not strong in js. But in my opinion you just have an empty string. You don't check for the existence of the link before writing.
Maybe something like this:
var request = require('sync-request');
var cheerio = require('cheerio');
var mysql = require('mysql');
var connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: '',
database: 'prsr'
});
var baseUrl = 'https://en.wikipedia.org';
var startUrl = 'https://en.wikipedia.org/wiki/Category:Video_games_with_3D_graphics';
var res = request('GET', startUrl);
var $ = cheerio.load(res.getBody());
$('#mw-pages').find('.mw-category-group').each(function (index) {
var fullUrl = baseUrl + $(this).find('a').attr('href');
var data = parsePage(fullUrl);
if (data) {
connection.query('INSERT INTO games SET ?', data, function (err, result) {
if (err) throw err; // выкидывает ошибку здесь
console.log(result.insertId);
}
});
});
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question