Code recognition, state machine, lexical analyzer and other buzzwords?

Question

C++ / C#

I have very vague ideas about how to solve my problem. There is some text editor, you need to make sure that all elements of this code are recognized by this editor. This is implemented in any framework. The framework can determine where a class, for example, starts and where it ends. On the forums, I was told about the finite state machine, and I still did not understand how it could work with this particular task. Since I see that this problem has already been solved in many projects, there is an assumption that somewhere there is a ready-made example.

Answer 1

If you specifically need the simplest syntax highlighting (as in your html example), then lexical analysis is enough for you . You need to select tokens in the text, and depending on the type of token, choose a color for highlighting. In different environments, this will be implemented differently - in Sublime, this is done through a special. a language for describing finite automata, regular expressions can be used in other editors, a lexical analyzer is written in general-purpose PL in Visual Studio.
Look at these tools, C/C++: Flex , ANTLR for dotnet languages, FsLex is also good (although it has F# syntax).
If you need more complex analysis (for example, substitutions depending on the context), see the answerDmitry Makarov is already a syntactic analysis, the next stage after the lexical one. Said Yacc is a classic Lex/Flex pair. There is also a more modern analogue compatible with Yacc - Bison . For dotnet, I can advise the same ANTLR, and FsYacc in pair with FsLex.

Answer 2

I understand you here: Parsing and, for example, here: yacc

Answer 3

On the forums, I was told about the finite state machine, and I still did not understand how it could work with this particular task.

If you don’t know what a finite automaton is (never did), then you won’t solve this problem (only with errors if). The state machine does not allow, for example, to select a keyword that is inside a comment, because it understands where the place is outside the comment, and where the place is inside the comment. Therefore, in such cases, regular expressions will not help either (as many people think).