How to parse a table from html using Scrapy?

Question

Python

There is an html page with nested tables

. The html table of interest looks like this:

<tr valign="top">
<td colspan="2" bgcolor="F2F2F2"><a href="/trk/fe/TEDIS_466.nsf/,DanaInfo=extranet2.de.tuv.com+Antragsdaten?OpenAgent&amp;U=9FE850F57F956872C1258366002703BB"><b>Abdellaoui, Sabrine</b></a></td>
<td bgcolor="F2F2F2">20.11.1995&nbsp;<img src="/trk/fe/TEDIS_466.nsf/,DanaInfo=extranet2.de.tuv.com+empty.gif?OpenImageResource" border="0" align="BOTTOM"></td>
<td bgcolor="F2F2F2"><b>0002906617</b></td>
<td bgcolor="F2F2F2"><b>477306</b></td>
</tr>
<tr valign="MIDDLE">
<td>&nbsp;&nbsp;&nbsp;</td>
<td bgcolor="FFFFFF">Theorie Klasse B</td>
<td bgcolor="FFFFFF">26.02.2019</td>
<td bgcolor="FFFFFF">bezahlt</td>
<td bgcolor="FFFFFF">bestanden</td>
</tr>
<tr valign="MIDDLE">
<td>&nbsp;&nbsp;&nbsp;</td>
<td bgcolor="FFFFFF">Fahren Klasse B</td>
<td bgcolor="FFFFFF"></td>
<td bgcolor="FFFFFF">offen</td>
<td bgcolor="FFFFFF">noch offen</td>
</tr>
<tr valign="top">
<td colspan="2" bgcolor="F2F2F2"><a href="/trk/fe/TEDIS_466.nsf/,DanaInfo=extranet2.de.tuv.com+Antragsdaten?OpenAgent&amp;U=513F853FA943F430C1258397002C01AF"><b>Abdulkader, Scheima</b></a></td>
<td bgcolor="F2F2F2">01.08.1997&nbsp;<img src="/trk/fe/TEDIS_466.nsf/,DanaInfo=extranet2.de.tuv.com+empty.gif?OpenImageResource" border="0" align="BOTTOM"></td>
<td bgcolor="F2F2F2"><b>0002929729</b></td>
<td bgcolor="F2F2F2"><b>481192</b></td>
</tr>
<tr valign="MIDDLE">
<td>&nbsp;&nbsp;&nbsp;</td>
<td bgcolor="FFFFFF">Theorie Klasse B</td>
<td bgcolor="FFFFFF">14.03.2019</td>
<td bgcolor="FFFFFF">bezahlt</td>
<td bgcolor="FFFFFF">bestanden</td>
</tr>
<tr valign="MIDDLE">
<td>&nbsp;&nbsp;&nbsp;</td>
<td bgcolor="FFFFFF">Fahren Klasse B</td>
<td bgcolor="FFFFFF"></td>
<td bgcolor="FFFFFF">bezahlt</td>
<td bgcolor="FFFFFF">noch offen</td>
</tr>

Question:
How to parse data so that the string tr valign="top" and all the strings tr valign="MIDDLE" up to the next tr valign="top" get into one dictionary?
Sample output that should be:
[
{"name": "Abdulkader, Scheima", "gebDatum": "11/20/1995", "aktenzeichen": "0002906617", "antragNr": "477306", "prufung": [ "Theorie Klasse B", "Fahren Klasse B"], "prufungsdatum": ["26.02.2019", "14.03.2019"], "Zahlung": ["bezahlt", "offen"], "Ergebnis": [ "bestanden", "noch offen"]}
]
Now I'm manually parsing all the cells,

def parse(self, response):
        all = response.xpath('//table[position()>2]/tbody')
        for fld in all:
            Item = DataItem()
            names = fld.xpath('//tr/td/a/b/text()').extract()
            bday = fld.xpath('//tr[@valign="top"]/td[position()=2]/text()').getall()
            pass_number = fld.xpath('//tr/td[position()=3]/b/text()').getall()
            order_number = fld.xpath('//tr/td[position()=4]/b/text()').getall()
            exam_name = fld.xpath('//tr[@valign="MIDDLE"]/td[position()=2]/text()').getall()
            exam_date = fld.xpath('//tr[@valign="MIDDLE"]/td[position()=3]/text()').getall()
            exam_pay = fld.xpath('//tr[@valign="MIDDLE"]/td[position()=4]/text()').getall()
            exam_result =fld.xpath('//tr[@valign="MIDDLE"]/td[position()=5]/text()').getall()
            Item['name'] = names[1]
            Item['gebDatum'] = bday[1][0:10]
            Item['aktenzeichen'] = pass_number[1]
            Item['antragNr'] = order_number[1]
            Item['prufung'] = exam_name[0:2]
            Item['prufungsdatum'] = exam_date[0:2]
            Item['Zahlung'] = exam_pay[0:2]
            Item['Ergebnis'] = exam_result[0:2]
            yield Item

Answer 1

pytablereader