O
O
Orkhan Hasanli2020-07-22 18:02:04
Java
Orkhan Hasanli, 2020-07-22 18:02:04

How to correctly generate DTO from json for subsequent saving in the database?

Good day!
There are json / geojson files of huge sizes (from 120 mb to 2 GB in weight).
It is necessary to form a DTO based on the json file in order to subsequently serialize and save it to the database using jackson.
Usually, I use an online service to generate DTO - www.jsonschema2pojo.org
But it has a limit of about 50,000 characters. Accordingly, you have to split the file into parts and each time copy / delete duplicate fields, etc. At the moment, we managed to get about 1000 fields, and apparently this is not the end. I can, of course, ignore the fields using, @JsonIgnoreProperties(ignoreUnknown = true)at the class level, but I would like to map all the fields. This begs the question, what are the tools for working with large amounts of data?
Tried the following options:
www.jsonschema2pojo.org online service, cli, maven plugin, gradle plugin. Only the online version generates correctly, but has limitations. cli & maven, gradle plugin do not form dto or form incorrectly.
I tried this tool https://github.com/astav/JsonToJava JsonToJava, it also forms incorrectly.
At the moment, one idea comes to mind:
read the file line by line using Java, use Selenium + webdriver to copy it into the browser to the above online service, then get all the fields and remove duplicate fields. Are there better solutions to the issue?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
Y
Yurii Nekrasov, 2020-07-30
@azerphoenix

It is necessary to form a DTO based on the json file in order to subsequently serialize and save it to the database using jackson

I don't quite understand the question. Do you mean how to create the objects themselves? That is, in order not to create classes with fields with handles?
Maybe this link will help you https://github.com/astav/JsonToJava

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question